The Elasticsearch module has been tested with versions 5, 6 and 7.

Its behavior will differ based on the version, as to adapt itself to the changes in the Elasticsearch modeling (namely, the removal of mapping types). For instance, when cataloging on a version 5 instance, the dataset is the mapping type; whereas for versions 6 and 7, the dataset is the index. However, version 6 still displays the type, a unique value but whose name might have been customized.

Since scanner version 26.9, the Elasticsearch plugin can be downloaded here: Zeenea Connector Downloads.

For more information on how to install a plugin, please refer to the following article: Installing and Configuring Connectors as a Plugin.

Creating and configuring connectors is done through a dedicated configuration file located in the /connections folder of the relevant scanner.

Read more: Managing Connections

In order to establish a connection with Elasticsearch, specifying the following parameters in the dedicated file is required:

The connector is able to manage virtual indexes that have been partitioned on multiple real indexes.

A Partitioned Virtual Index is a collection of indexes containing the same dataset and whose use the following naming convention:

For instance, in the case of the Elasticsearch indexation of the Cogip application logs, best practice is to split the indexes by time frames (for example, on a daily basis). The indexes names will then be:

The share part is log-cogip, whereas the part representing the partition can be described with the rational expression:

\d{4}-\d{2}-\d{2}$

Declaring the rational expression above will be enough for Zeenea to take into account the partitioned virtual indexes.

During inventory, Zeenea will replace the first occurrence of this model with a star. All indexes with the same name structure will be considered as being a part of the same partitioned virtual index.

In our example above, all three partitions will be read as a unique dataset named log-cogip-*.

If multiple models are used, all corresponding rational expressions must be declared and separated with spaces. The first of these expressions that returns a result will be used.

For instance, if the following indexes are used:

By defining the setting Index Partition Pattern with the value: \d{4}-\d{2}-\d{2}$ \d{2}$, Zeenea will display the following datasets:

However, be mindful of the pattern order: if above expression is replaced with \d{2}$ \d{4}-\d{2}-\d{2}$, the first option, replacing the last two digits with a star, will be used, and the datasets displayed by Zeenea will be the following:

The pattern does not have to be at the end of a string; when using the rational expression \d{4}-\d{2}-\d{2} (note the absence of the dollar sign) on the following indexes:

will return the unique dataset copgip-$-log.

On the other hand, an index named log-cogip-2020-06-01_2020-06-07 will be identified as log-cogip-*_2020-06-07 when using an Index Partition Pattern value of \d{4}-\d{2}-\d{2}, and log-cogip-2020-06-01 with a value of \d{4}-\d{2}-\d{2}$.

It is then strongly recommended to use a rational expression as specific as possible to avoid unexpected results.

The Elasticsearch user being used for the connection must have sufficient permissions to extract metadata and collect data statistics.

The following requests are necessary:

For Elasticsearch version 5, the inventory collects the names of indexes and their types. The dataset corresponds to the type.

For Elasticsearch version 6, the inventory collects the names of indexes and their unique type.

From version 7 onwards, the inventory collects the names of indexes.

Indexes are unpartitioned using the procedure described in Partitioned Virtual Indexes.

Based on the version of Elasticsearch, it is one dataset per index (v6 onwards) or one dataset per type (v5).

An identification key is associated with each object in the catalog. In the case of the object being created by a connector, the connector builds it.

Read more: Identification Keys