--- title: Elastic Integration filter plugin v0.0.1 description: Plugin version: v0.0.1, Released on: 2023-04-14, Changelog. For other versions, see the overview list. To learn more about Logstash, see the Logstash... url: https://www.elastic.co/docs/reference/logstash/versioned-plugins/v0-0-1-plugins-filters-elastic_integration products: - Logstash applies_to: - Elastic Stack: Generally available --- # Elastic Integration filter plugin v0.0.1 - Plugin version: v0.0.1 - Released on: 2023-04-14 - [Changelog](https://github.com/elastic/logstash-filter-elastic_integration/blob/v0.0.1/CHANGELOG.md) For other versions, see the [overview list](https://www.elastic.co/docs/reference/logstash/versioned-plugins/filter-elastic_integration-index). To learn more about Logstash, see the [Logstash Reference](https://www.elastic.co/guide/en/logstash/current/index.html). ## Getting help For questions about the plugin, open a topic in the [Discuss](http://discuss.elastic.co) forums. For bugs or feature requests, open an issue in [Github](https://github.com/elastic/logstash-filter-elastic_integration). For the list of Elastic supported plugins, please consult the [Elastic Support Matrix](https://www.elastic.co/support/matrix#matrix_logstash_plugins). **Elastic Enterprise License** Use of this plugin requires an active Enterprise Elastic License. ## Description Using this filter you can process Elastic integrations powered by Elasticsearch Ingest Node in Logstash. Once configured to point to an Elasticsearch cluster, events processed by this filter will first resolve their effective data-stream, and determine from Elasticsearch which ingest pipeline to run. The ingest pipeline will be run inside Logstash without transmitting the event to Elasticsearch. Events that are successfully handled by their ingest pipeline will have `[@metadata][target_ingest_pipeline]` set to `+_none+` so that any downstream Elasticsearch output in the Logstash pipeline will avoid running the event’s default pipeline *again* in Elasticsearch. Some multi-pipeline configurations such as logstash-to-logstash over http(s) do not maintain the state of `[@metadata]` fields. In these setups, you may need to explicitly configure your downstream pipeline’s Elasticsearch output with `+pipeline => "_none"+` to avoid re-running the default pipeline. Events that *fail* ingest pipeline processing will be tagged with `_ingest_pipeline_failure`, and their `[@metadata][_ingest_pipeline_failure]` will be populated with details as a key/value map. This plugin requires minimum Java 17 and Logstash 8.7.0 versions. **Technology Preview** This Elastic Integration filter plugin is part of a *Technology Preview*, which means that both configuration options and implementation details are subject to change in minor releases without being preceded by deprecation warnings. Before upgrading this plugin or Logstash itself, please pay special attention to the changelogs to avoid being caught by surprise. ## Minimum Configuration You will need to configure this plugin to connect to Elasticsearch, and may need to also need to provide local GeoIp databases. ``` filter { elastic_integration { cloud_id => "YOUR_CLOUD_ID_HERE" cloud_auth => "YOUR_CLOUD_AUTH_HERE" geoip_database_directory => "/etc/your/geoip-databases" } } ``` Read on for a guide to configuration, or jump to the [complete list of configuration options](#v0.0.1-plugins-filters-elastic_integration-options). ## Connecting to Elasticsearch This plugin communicates with Elasticsearch to identify which ingest pipeline should be run for a given event, and to retrieve the ingest pipeline definitions themselves. You must configure this plugin to point to Elasticsearch using exactly one of: - A Cloud Id (see [`cloud_id`](#v0.0.1-plugins-filters-elastic_integration-cloud_id)) - A list of one or more host URLs (see [`hosts`](#v0.0.1-plugins-filters-elastic_integration-hosts)) Communication will be made securely over SSL unless you explicitly configure this plugin otherwise. You may need to configure how this plugin establishes trust of the server that responds, and will likely need to configure how this plugin presents its own identity or credentials. ### SSL Trust Configuration When communicating over SSL, this plugin fully-validates the proof-of-identity presented by Elasticsearch using the system trust store. You can provide an *alternate* source of trust with one of: - A PEM-formatted list of trusted certificate authorities (see [`ssl_certificate_authorities`](#v0.0.1-plugins-filters-elastic_integration-ssl_certificate_authorities)) - A JKS- or PKCS12-formatted Keystore containing trusted certificates (see [`ssl_truststore_path`](#v0.0.1-plugins-filters-elastic_integration-ssl_truststore_path)) You can also configure which aspects of the proof-of-identity are verified (see [`ssl_verification_mode`](#v0.0.1-plugins-filters-elastic_integration-ssl_verification_mode)). ### SSL Identity Configuration When communicating over SSL, you can also configure this plugin to present a certificate-based proof-of-identity to the Elasticsearch cluster it connects to using one of: - A PKCS8 Certificate/Key pair (see [`ssl_certificate`](#v0.0.1-plugins-filters-elastic_integration-ssl_certificate)) - A JKS- or PKCS12-formatted Keystore (see [`ssl_keystore_path`](#v0.0.1-plugins-filters-elastic_integration-ssl_keystore_path)) ### Request Identity You can configure this plugin to present authentication credentials to Elasticsearch in one of several ways: - ApiKey: (see [`api_key`](#v0.0.1-plugins-filters-elastic_integration-api_key)) - Cloud Auth: (see [`cloud_auth`](#v0.0.1-plugins-filters-elastic_integration-cloud_auth)) - HTTP Basic Auth: (see [`auth_basic_username`](#v0.0.1-plugins-filters-elastic_integration-auth_basic_username) and [`auth_basic_password`](#v0.0.1-plugins-filters-elastic_integration-auth_basic_password)) Your request credentials are only as secure as the connection they are being passed over. They provide neither privacy nor secrecy on their own, and can easily be recovered by an adversary when SSL is disabled. ## Supported Ingest Processors This filter can run Elasticsearch Ingest Node pipelines that are *wholly* comprised of the supported subset of processors. It has access to the Painless and Mustache scripting engines where applicable: | Source | Processor | Caveats | |-------------------|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------| | **Ingest Common** | `append` | *none* | | | `bytes` | *none* | | | `communityid` | *none* | | | `convert` | *none* | | | `csv` | *none* | | | `date` | *none* | | | `dateindexname` | *none* | | | `dissect` | *none* | | | `dotexpander` | *none* | | | `drop` | *none* | | | `fail` | *none* | | | `fingerprint` | *none* | | | `foreach` | *none* | | | `grok` | *none* | | | `gsub` | *none* | | | `htmlstrip` | *none* | | | `join` | *none* | | | `json` | *none* | | | `keyvalue` | *none* | | | `lowercase` | *none* | | | `networkdirection` | *none* | | | `pipeline` | resolved pipeline *must* be wholly-composed of supported processors | | | `redact` | *none* | | | `registereddomain` | *none* | | | `remove` | *none* | | | `rename` | *none* | | | `script` | *none* | | | `set` | *none* | | | `sort` | *none* | | | `split` | *none* | | | `trim` | *none* | | | `uppercase` | *none* | | | `uriparts` | *none* | | | `urldecode` | *none* | | **GeoIp** | `geoip` | Requires local MaxMind GeoIp databases [`geoip_database_directory`](#v0.0.1-plugins-filters-elastic_integration-geoip_database_directory) | ### Field Mappings During execution, the Ingest pipeline works with a temporary mutable *view* of the Logstash event that re-shapes some Logstash-reserved fields into their expected IngestDocument field names and object-types. Changes to the IngestDocument will be reflected in the resulting Logstash Event, including safely mapping these reserved fields *back* from the IngestDocument reserved field to the Logstash reserved field counterpart. | Logstash Field | IngestDocument Field | Conflict Handling | |-------------------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `@timestamp` | `_ingest.timestamp` | when ingest processing *also* sets a top-level `@timestamp` field, it will be made available via the Event’s `_@timestamp` field | | `@version` | `_version` | when ingest processing *also* sets a top-level `@version` field in the source, it will be made available via the Event’s `_@version` field | | `@metadata` | `@metadata` | when ingest processing replaces the top-level `@metadata` map with an object that is not a string-object map, it will be made available via the Event’s `_@metadata` field. | | `tags` | `tags` | when ingest processing produces a top-level `tags` field that is not a collection of strings, it will be made available via the event’s `_tags` field. | | *everything else* | *in-place, as-structured* | only minimal type conversions are done | ## Elastic Integration Filter Configuration Options This plugin supports the following configuration options plus the [Common options](#v0.0.1-plugins-filters-elastic_integration-common-options) described later. | Setting | Input type | Required | |----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|----------| | [`api_key`](#v0.0.1-plugins-filters-elastic_integration-api_key) | [password](/docs/reference/logstash/plugins/value-types#password) | No | | [`auth_basic_password`](#v0.0.1-plugins-filters-elastic_integration-auth_basic_password) | [password](/docs/reference/logstash/plugins/value-types#password) | No | | [`auth_basic_username`](#v0.0.1-plugins-filters-elastic_integration-auth_basic_username) | [string](/docs/reference/logstash/plugins/value-types#string) | No | | [`cloud_auth`](#v0.0.1-plugins-filters-elastic_integration-cloud_auth) | [password](/docs/reference/logstash/plugins/value-types#password) | No | | [`cloud_id`](#v0.0.1-plugins-filters-elastic_integration-cloud_id) | [string](/docs/reference/logstash/plugins/value-types#string) | No | | [`hosts`](#v0.0.1-plugins-filters-elastic_integration-hosts) | [array](/docs/reference/logstash/plugins/value-types#array) | No | | [`ssl_certificate`](#v0.0.1-plugins-filters-elastic_integration-ssl_certificate) | [path](/docs/reference/logstash/plugins/value-types#path) | No | | [`ssl_certificate_authorities`](#v0.0.1-plugins-filters-elastic_integration-ssl_certificate_authorities) | [array](/docs/reference/logstash/plugins/value-types#array) | No | | [`ssl_enabled`](#v0.0.1-plugins-filters-elastic_integration-ssl_enabled) | [boolean](/docs/reference/logstash/plugins/value-types#boolean) | No | | [`ssl_key`](#v0.0.1-plugins-filters-elastic_integration-ssl_key) | [path](/docs/reference/logstash/plugins/value-types#path) | No | | [`ssl_keystore_password`](#v0.0.1-plugins-filters-elastic_integration-ssl_keystore_password) | [password](/docs/reference/logstash/plugins/value-types#password) | No | | [`ssl_keystore_path`](#v0.0.1-plugins-filters-elastic_integration-ssl_keystore_path) | [path](/docs/reference/logstash/plugins/value-types#path) | No | | [`ssl_key_passphrase`](#v0.0.1-plugins-filters-elastic_integration-ssl_key_passphrase) | [password](/docs/reference/logstash/plugins/value-types#password) | No | | [`ssl_truststore_path`](#v0.0.1-plugins-filters-elastic_integration-ssl_truststore_path) | [path](/docs/reference/logstash/plugins/value-types#path) | No | | [`ssl_truststore_password`](#v0.0.1-plugins-filters-elastic_integration-ssl_truststore_password) | [password](/docs/reference/logstash/plugins/value-types#password) | No | | [`ssl_verification_mode`](#v0.0.1-plugins-filters-elastic_integration-ssl_verification_mode) | [string](/docs/reference/logstash/plugins/value-types#string), one of `["full", "certificate", "none"]` | No | ### `api_key` - Value type is [password](/docs/reference/logstash/plugins/value-types#password) - There is no default value for this setting. The encoded form of an API key that is used to authenticate this plugin to Elasticsearch ### `auth_basic_password` - Value type is [password](/docs/reference/logstash/plugins/value-types#password) - There is no default value for this setting. - Required when request auth is configured with [`auth_basic_username`](#v0.0.1-plugins-filters-elastic_integration-auth_basic_username) A password when using HTTP Basic Authentication to connect to Elasticsearch. ### `auth_basic_username` - Value type is [string](/docs/reference/logstash/plugins/value-types#string) - There is no default value for this setting. - When present, [`auth_basic_password`](#v0.0.1-plugins-filters-elastic_integration-auth_basic_password) is also required. A user name when using HTTP Basic Authentication to connect to Elasticsearch. ### `cloud_auth` - Value type is [password](/docs/reference/logstash/plugins/value-types#password) - There is no default value for this setting. Cloud authentication string (":" format) is an alternative for the `auth_basic_username`/`auth_basic_password` pair and can be obtained from Elastic Cloud web console. ### `cloud_id` - Value type is [string](/docs/reference/logstash/plugins/value-types#string) - There is no default value for this setting. - Cannot be combined with `[`ssl_enabled`](#v0.0.1-plugins-filters-elastic_integration-ssl_enabled)⇒false`. Cloud Id, from the Elastic Cloud web console. When connecting with a Cloud Id, communication to Elasticsearch is secured with SSL. For more details, check out the [Logstash-to-Cloud documentation](https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html). ### `hosts` - Value type is a list of [uri](/docs/reference/logstash/plugins/value-types#uri)s - There is no default value for this setting. - Constraints: - When any URL contains a protocol component, all URLs must have the same protocol as each other. - `https`-protocol hosts use HTTPS and cannot be combined with [`+ssl_enabled => false+`](#v0.0.1-plugins-filters-elastic_integration-ssl_enabled). - `http`-protocol hosts use unsecured HTTP and cannot be combined with [`+ssl_enabled => true+`](#v0.0.1-plugins-filters-elastic_integration-ssl_enabled). - When any URL omits a port component, the default `9200` is used. - When any URL contains a path component, all URLs must have the same path as each other. A non-empty list of Elasticsearch hosts to connect. Examples: ``` `"127.0.0.1"` `["127.0.0.1:9200","127.0.0.2:9200"]` `["http://127.0.0.1"]` `["https://127.0.0.1:9200"]` `["https://127.0.0.1:9200/subpath"]` (If using a proxy on a subpath) ``` When connecting with a list of hosts, communication to Elasticsearch is secured with SSL unless configured otherwise. Disabling SSL is dangerous The security of this plugin relies on SSL to avoid leaking credentials and to avoid running illegitimate ingest pipeline definitions. There are two ways to disable SSL: - Provide a list of `http`-protocol hosts - Set [`+ssl_enabled => false+`](#v0.0.1-plugins-filters-elastic_integration-ssl_enabled) ### `ssl_certificate` - Value type is [path](/docs/reference/logstash/plugins/value-types#path) - There is no default value for this setting. - When present, [`ssl_key`](#v0.0.1-plugins-filters-elastic_integration-ssl_key) and [`ssl_key_passphrase`](#v0.0.1-plugins-filters-elastic_integration-ssl_key_passphrase) are also required. - Cannot be combined with configurations that disable SSL Path to a PEM-encoded certificate or certificate chain with which to identify this plugin to Elasticsearch. ### `ssl_certificate_authorities` - Value type is a list of [path](/docs/reference/logstash/plugins/value-types#path)s - There is no default value for this setting. - Cannot be combined with configurations that disable SSL - Cannot be combined with `[`ssl_verification_mode`](#v0.0.1-plugins-filters-elastic_integration-ssl_verification_mode)⇒none`. One or more PEM-formatted files defining certificate authorities. This setting can be used to *override* the system trust store for verifying the SSL certificate presented by Elasticsearch. ### `ssl_enabled` - Value type is [boolean](/docs/reference/logstash/plugins/value-types#boolean) - There is no default value for this setting. Secure SSL communication to Elasticsearch is enabled unless: - it is explicitly disabled with `+ssl_enabled => false+`; OR - it is implicitly disabled by providing `http`-protocol [`hosts`](#v0.0.1-plugins-filters-elastic_integration-hosts). Specifying `+ssl_enabled => true+` can be a helpful redundant safeguard to ensure this plugin cannot be configured to use non-ssl communication. ### `ssl_key` - Value type is [path](/docs/reference/logstash/plugins/value-types#path) - There is no default value for this setting. - Required when connection identity is configured with [`ssl_certificate`](#v0.0.1-plugins-filters-elastic_integration-ssl_certificate) - Cannot be combined with configurations that disable SSL A path to a PKCS8-formatted SSL certificate key. ### `ssl_keystore_password` - Value type is [password](/docs/reference/logstash/plugins/value-types#password) - There is no default value for this setting. - Required when connection identity is configured with [`ssl_keystore_path`](#v0.0.1-plugins-filters-elastic_integration-ssl_keystore_path) - Cannot be combined with configurations that disable SSL Password for the [`ssl_keystore_path`](#v0.0.1-plugins-filters-elastic_integration-ssl_keystore_path) ### `ssl_keystore_path` - Value type is [path](/docs/reference/logstash/plugins/value-types#path) - There is no default value for this setting. - When present, [`ssl_keystore_password`](#v0.0.1-plugins-filters-elastic_integration-ssl_keystore_password) is also required. - Cannot be combined with configurations that disable SSL A path to a JKS- or PKCS12-formatted keystore with which to identify this plugin to Elasticsearch. ### `ssl_key_passphrase` - Value type is [password](/docs/reference/logstash/plugins/value-types#password) - There is no default value for this setting. - Required when connection identity is configured with [`ssl_certificate`](#v0.0.1-plugins-filters-elastic_integration-ssl_certificate) - Cannot be combined with configurations that disable SSL A password or passphrase of the [`ssl_key`](#v0.0.1-plugins-filters-elastic_integration-ssl_key). ### `ssl_truststore_path` - Value type is [path](/docs/reference/logstash/plugins/value-types#path) - There is no default value for this setting. - When present, [`ssl_truststore_password`](#v0.0.1-plugins-filters-elastic_integration-ssl_truststore_password) is required. - Cannot be combined with configurations that disable SSL - Cannot be combined with `[`ssl_verification_mode`](#v0.0.1-plugins-filters-elastic_integration-ssl_verification_mode)⇒none`. A path to JKS- or PKCS12-formatted keystore where trusted certificates are located. This setting can be used to *override* the system trust store for verifying the SSL certificate presented by Elasticsearch. ### `ssl_truststore_password` - Value type is [password](/docs/reference/logstash/plugins/value-types#password) - There is no default value for this setting. - Required when connection trust is configured with [`ssl_truststore_path`](#v0.0.1-plugins-filters-elastic_integration-ssl_truststore_path) - Cannot be combined with configurations that disable SSL Password for the [`ssl_truststore_path`](#v0.0.1-plugins-filters-elastic_integration-ssl_truststore_path). ### `ssl_verification_mode` - Value type is [string](/docs/reference/logstash/plugins/value-types#string) - There is no default value for this setting. - Cannot be combined with configurations that disable SSL Level of verification of the certificate provided by Elasticsearch. SSL certificates presented by Elasticsearch are fully-validated by default. - Available modes: - `none`: performs no validation, implicitly trusting any server that this plugin connects to (insecure) - `certificate`: validates the server-provided certificate is signed by a trusted certificate authority and that the server can prove possession of its associated private key (less secure) - `full` (default): performs the same validations as `certificate` and also verifies that the provided certificate has an identity claim matching the server we are attempting to connect to (most secure) ### `geoip_database_directory` - Value type is [path](/docs/reference/logstash/plugins/value-types#path) - There is no default value for this setting. When using the GeoIp processor, this plugin does *not* have access to the GeoIp databases that are a part of the Elasticsearch cluster, and must be configured with locally-available database files. Databases are registered by file name, and most integrations rely on databases being present named: - `GeoLite2-ASN.mmdb` - `GeoLite2-City.mmdb` - `GeoLite2-Country.mmdb` This plugin will discover any regular file with the `.mmdb` suffix in the provided directory, and expects the files it finds to be in the MaxMind DB format. ## Common options These configuration options are supported by all filter plugins: | Setting | Input type | Required | |--------------------------------------------------------------------------------|-----------------------------------------------------------------|----------| | [`add_field`](#v0.0.1-plugins-filters-elastic_integration-add_field) | [hash](/docs/reference/logstash/plugins/value-types#hash) | No | | [`add_tag`](#v0.0.1-plugins-filters-elastic_integration-add_tag) | [array](/docs/reference/logstash/plugins/value-types#array) | No | | [`enable_metric`](#v0.0.1-plugins-filters-elastic_integration-enable_metric) | [boolean](/docs/reference/logstash/plugins/value-types#boolean) | No | | [`id`](#v0.0.1-plugins-filters-elastic_integration-id) | [string](/docs/reference/logstash/plugins/value-types#string) | No | | [`periodic_flush`](#v0.0.1-plugins-filters-elastic_integration-periodic_flush) | [boolean](/docs/reference/logstash/plugins/value-types#boolean) | No | | [`remove_field`](#v0.0.1-plugins-filters-elastic_integration-remove_field) | [array](/docs/reference/logstash/plugins/value-types#array) | No | | [`remove_tag`](#v0.0.1-plugins-filters-elastic_integration-remove_tag) | [array](/docs/reference/logstash/plugins/value-types#array) | No | ### `add_field` - Value type is [hash](/docs/reference/logstash/plugins/value-types#hash) - Default value is `{}` If this filter is successful, add any arbitrary fields to this event. Field names can be dynamic and include parts of the event using the `%{field}`. Example: ``` filter { elastic_integration { add_field => { "foo_%{somefield}" => "Hello world, from %{host}" } } } ``` ``` # You can also add multiple fields at once: filter { elastic_integration { add_field => { "foo_%{somefield}" => "Hello world, from %{host}" "new_field" => "new_static_value" } } } ``` If the event has field `"somefield" == "hello"` this filter, on success, would add field `foo_hello` if it is present, with the value above and the `%{host}` piece replaced with that value from the event. The second example would also add a hardcoded field. ### `add_tag` - Value type is [array](/docs/reference/logstash/plugins/value-types#array) - Default value is `[]` If this filter is successful, add arbitrary tags to the event. Tags can be dynamic and include parts of the event using the `%{field}` syntax. Example: ``` filter { elastic_integration { add_tag => [ "foo_%{somefield}" ] } } ``` ``` # You can also add multiple tags at once: filter { elastic_integration { add_tag => [ "foo_%{somefield}", "taggedy_tag"] } } ``` If the event has field `"somefield" == "hello"` this filter, on success, would add a tag `foo_hello` (and the second example would of course add a `taggedy_tag` tag). ### `enable_metric` - Value type is [boolean](/docs/reference/logstash/plugins/value-types#boolean) - Default value is `true` Disable or enable metric logging for this specific plugin instance by default we record all the metrics we can, but you can disable metrics collection for a specific plugin. ### `id` - Value type is [string](/docs/reference/logstash/plugins/value-types#string) - There is no default value for this setting. Add a unique `ID` to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type, for example, if you have 2 elastic_integration filters. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs. ``` filter { elastic_integration { id => "ABC" } } ``` ### `periodic_flush` - Value type is [boolean](/docs/reference/logstash/plugins/value-types#boolean) - Default value is `false` Call the filter flush method at regular interval. Optional. ### `remove_field` - Value type is [array](/docs/reference/logstash/plugins/value-types#array) - Default value is `[]` If this filter is successful, remove arbitrary fields from this event. Fields names can be dynamic and include parts of the event using the %{field} Example: ``` filter { elastic_integration { remove_field => [ "foo_%{somefield}" ] } } ``` ``` # You can also remove multiple fields at once: filter { elastic_integration { remove_field => [ "foo_%{somefield}", "my_extraneous_field" ] } } ``` If the event has field `"somefield" == "hello"` this filter, on success, would remove the field with name `foo_hello` if it is present. The second example would remove an additional, non-dynamic field. ### `remove_tag` - Value type is [array](/docs/reference/logstash/plugins/value-types#array) - Default value is `[]` If this filter is successful, remove arbitrary tags from the event. Tags can be dynamic and include parts of the event using the `%{field}` syntax. Example: ``` filter { elastic_integration { remove_tag => [ "foo_%{somefield}" ] } } ``` ``` # You can also remove multiple tags at once: filter { elastic_integration { remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"] } } ``` If the event has field `"somefield" == "hello"` this filter, on success, would remove the tag `foo_hello` if it is present. The second example would remove a sad, unwanted tag as well.