Data source settings reference for ES|QL Data Federation

Data source settings control how Elasticsearch connects and authenticates to an external storage system. Add settings to the settings object when you create or update a data source, or fill in the matching fields in the Connect data source flyout in Kibana. The available settings depend on the data source type.

Warning

This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.

Amazon S3

The following settings apply to data sources with type set to s3.

Connection settings

These settings control which endpoint Elasticsearch sends S3 requests to and how it addresses buckets.

endpoint

An Amazon S3 endpoint override, for example https://s3.us-east-1.amazonaws.com.

Default: None. The endpoint is resolved from the region.
Valid values:
- The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. An absolute URL naming a supported AWS S3 endpoint. Refer to S3 endpoint requirements.
- The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Experimental in 9.5 This functionality is experimental and is not ready for production usage. Experimental features may change or be removed at any time. Elastic will work to fix any issues, but experimental features are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Any endpoint URL. The value isn't validated.
Related: addressing_style, region

Omit endpoint to resolve the endpoint from the region, which is the recommended configuration.

Stack Planned S3 endpoint requirements

The endpoint value must be an absolute https URL naming a supported AWS S3 endpoint. Elasticsearch accepts the following endpoint forms. Every AWS partition supports the regional, historical, and VPC interface forms, and only the commercial partition supports the global form:

Regional: https://s3.us-east-1.amazonaws.com
Historical: https://s3-us-west-2.amazonaws.com
VPC interface: https://bucket.vpce-0a1b2c3d.s3.us-east-1.vpce.amazonaws.com
Global: https://s3.amazonaws.com

A regional endpoint must name a region that your Elasticsearch version knows about. Elasticsearch rejects a region that AWS added after that release until you upgrade, or until a node permits its host with esql.external.allowed_endpoint_hosts.

Note

https://s3.amazonaws.com has no region. When you set endpoint, the AWS SDK stops following cross-region redirects, so this global endpoint only reaches us-east-1 buckets. Other regions get an error. Omit endpoint to let the AWS SDK resolve the correct regional endpoint from the region setting.

Elasticsearch rejects every other AWS endpoint family, including FIPS endpoints, dual-stack endpoints, transfer acceleration, access points, object lambda, Outposts, the account-level control plane, the legacy s3-external-1 alias, and S3 Express. It also rejects a bucket-qualified endpoint such as https://mybucket.s3.us-east-1.amazonaws.com. Name the regional endpoint instead, and let the bucket come from the dataset. It also rejects plain http, a value without a scheme, and a host that the URL syntax doesn't allow, such as one with an underscore or a non-numeric port.

A node can permit additional hosts with the esql.external.allowed_endpoint_hosts node setting, a list of host:port patterns in elasticsearch.yml that defaults to empty. A host it names is also accepted over plain http for endpoint. The sts_endpoint setting always requires https.

Warning

A data source created before these endpoint restrictions were introduced keeps working for queries, but updating it requires an endpoint that meets these requirements.

addressing_style The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

The URL addressing style for S3 requests.

Default: auto
Valid values:
- auto: Uses path-style when endpoint is set, and the AWS SDK default otherwise.
- path: Always uses path-style.
- virtual_hosted: Lets the AWS SDK decide. Bare-IP endpoints fall back to path-style.
Related: endpoint

Because auto resolves to path-style whenever endpoint is set, set virtual_hosted if reads through a VPC interface endpoint fail with an addressing error.

region The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned for deprecation This functionality is planned to be deprecated in a future Elastic Stack update. Subject to changes. Experimental in 9.5 This functionality is experimental and is not ready for production usage. Experimental features may change or be removed at any time. Elastic will work to fix any issues, but experimental features are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

The AWS region used for the S3 client.

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. The region setting on a data source is deprecated and has no effect. Set region in the dataset settings instead, or omit it to let Elasticsearch detect the region automatically. When no endpoint is set, the AWS SDK redirects transparently. When one is set, Elasticsearch issues a HeadBucket probe on the first request and caches the discovered region. The cache is cleared after a few minutes without requests, and the next request discovers the region again.

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Experimental in 9.5 This functionality is experimental and is not ready for production usage. Experimental features may change or be removed at any time. Elastic will work to fix any issues, but experimental features are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Set region on the data source. Datasets don't accept a region setting.

Authentication settings

These settings select the authentication model and supply the values it needs. For the AWS Identity and Access Management (IAM) permissions the configured identity needs, refer to grant read access in Amazon S3.

auth

The authentication model the data source uses.

Default: auto
Valid values:
- auto: Infers the model from the other settings. Federated identity settings such as role_arn select federated_identity, and access_key with secret_key selects static_credentials. A data source with both, or with neither, is rejected. auto never selects anonymous or managed_identity, so set those explicitly.
- anonymous
- static_credentials
- managed_identity
- federated_identity

The Requires line on each of the following settings names the model the setting belongs to. When auth is auto or omitted, it refers to the model Elasticsearch infers.

access_key

The AWS access key ID.

Default: None
Requires: auth set to static_credentials
Related: secret_key, session_token

secret_key

The AWS secret access key.

Default: None
Requires: auth set to static_credentials
Related: access_key, session_token

session_token

The AWS session token for temporary security credentials issued by AWS Security Token Service (STS).

Default: None
Requires: auth set to static_credentials, with access_key and secret_key from the same temporary credentials
Related: access_key, secret_key

Temporary credentials expire. When they do, requests from the data source fail until you update all three values with new credentials.

role_arn

The Amazon Resource Name (ARN) of the IAM role Elasticsearch assumes through STS.

Default: None. Required when auth is set to federated_identity.
Requires: auth set to federated_identity
Related: jwt_audience, role_session_name, sts_endpoint, sts_region

jwt_audience

Overrides the JWT audience claim sent to STS.

Default: sts.amazonaws.com
Requires: auth set to federated_identity
Related: role_arn

role_session_name

A label for the assumed-role session.

Default: elasticsearch-esql-datasource
Requires: auth set to federated_identity
Related: role_arn

sts_endpoint The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

An STS endpoint override, for example https://sts.us-east-1.amazonaws.com.

Default: None
Valid values: An absolute https URL that meets the same S3 endpoint requirements as endpoint, but for STS hosts
Requires: auth set to federated_identity
Related: sts_region

Any host permitted through esql.external.allowed_endpoint_hosts receives the node's OIDC token, so only add hosts on trusted network paths.

sts_region

The AWS region of the STS endpoint.

Default:
- The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. The dataset's region setting, or us-east-1 if the dataset has no region
- The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Experimental in 9.5 This functionality is experimental and is not ready for production usage. Experimental features may change or be removed at any time. Elastic will work to fix any issues, but experimental features are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. The data source's region setting, or us-east-1 if no region is set
Requires: auth set to federated_identity
Related: sts_endpoint, region

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. A region set on the data source no longer applies to STS. If a federated_identity data source relied on it, set sts_region on the data source, or region on each dataset, to keep calling STS in that region.