Amazon S3 data sources for ES|QL Data Federation

An s3 data source reads files from Amazon S3. This page compares the authentication models, lists the AWS Identity and Access Management (IAM) permissions Elasticsearch needs to read your data, and explains how to troubleshoot access errors. For step-by-step setup, follow the guide for your authentication model.

Warning

This feature is experimental. It is not intended for production use and there are no guarantees around performance, scale, or stability in this release.

Choose an authentication model

Each authentication model has its own setup guide:

Model	`auth` value	Setup
Static credentials	`static_credentials`	Connect with static credentials
Federated identity	`federated_identity`	Connect with federated identity
Anonymous	`anonymous`	Quickstart
Managed identity	`managed_identity`	Authentication models

For a description of each model and where it's available, refer to authentication models. For every setting an s3 data source accepts, refer to the data source settings reference.

Grant read access

Elasticsearch reads your data as the IAM identity the data source uses: the IAM user that owns the access key, the IAM role that Elasticsearch assumes, or the node's own role. Attach a policy to that identity that allows the following actions:

Action	Resource	When it's required
`s3:GetObject`	The objects the dataset reads, for example `arn:aws:s3:::<bucket-name>/<path>/*`	Always
`s3:ListBucket`	The bucket, for example `arn:aws:s3:::<bucket-name>`	When a dataset's resource is a prefix or a glob pattern rather than a single file
`kms:Decrypt`	The AWS Key Management Service (AWS KMS) key that encrypts the objects	When the objects use server-side encryption with AWS KMS keys (SSE-KMS)

Scope the s3:GetObject resource to the prefixes your datasets use to grant the least access needed. When you add a dataset that reads from a new bucket or prefix, extend the policy to cover it. For a complete policy and AWS CLI commands, refer to the setup guide for your authentication model.

For anonymous data sources, the bucket policy must allow public access to the same actions.

Check access

To confirm the data source can read a dataset, run a query against the dataset. Creating a dataset doesn't contact S3.

Troubleshoot access errors

Elasticsearch doesn't check permissions when you create a dataset. A missing permission shows up as an error the first time you query the dataset:

Access denied reading [<file>] The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.: The identity can't read an object. Allow s3:GetObject on the object's path. If the message names a kms: action, the object is encrypted with a KMS key the identity can't use. Allow that action on the key.
Access denied listing objects in the configured path The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.: The identity can't list the bucket. Allow s3:ListBucket on the bucket, or change the dataset's resource to an exact file path.
Access denied listing objects in bucket [<bucket>] with prefix [<prefix>] The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Experimental in 9.5 This functionality is experimental and is not ready for production usage. Experimental features may change or be removed at any time. Elastic will work to fix any issues, but experimental features are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.: The identity can't list the bucket. Allow s3:ListBucket on the bucket.