Create and manage datasets with the Elasticsearch API

Datasets are managed under the /_query/dataset endpoint. All dataset operations require the index manage privilege on the dataset name, or a fine-grained dataset privilege. Refer to manage credentials and privileges for details.

This page is a quick reference for each operation. For the complete request and response details, follow the API reference links in the table.

PUT /_query/dataset/{name} creates a new dataset or replaces an existing one entirely.

A dataset cannot have the same name as an existing index, data stream, alias, or view, because dataset names share the same namespace. Dataset names must be lowercase and cannot begin with -, _, or +.

The optional description can be at most 1,000 characters long.

Stack Planned The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Planned We plan to add this functionality in a future Elastic Stack update. Subject to changes. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. For restrictions on S3 bucket names, aliases, and Amazon Resource Names (ARNs), refer to Amazon S3 resource restrictions.

PUT /_query/dataset/access_logs
{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}

curl -X PUT "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
  "data_source": "prod_s3_logs",
  "resource": "s3://logs-bucket/access/**/*.parquet",
  "description": "Production access logs",
  "settings": {
    "partition_detection": "hive"
  }
}'

Elasticsearch validates setting values when you register the dataset, rather than waiting until the first query. A value that isn't supported, such as a multi-character delimiter, an unknown encoding, or a segment_size below the minimum, returns a 400 error that identifies the setting.

After creating a dataset, verify the inferred schema by checking its field mappings.

GET /_query/dataset/{name} retrieves a dataset by name.

GET /_query/dataset/access_logs

curl -X GET "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}"

GET /_query/dataset returns all registered datasets.

GET /_query/dataset

curl -X GET "${ELASTICSEARCH_URL}/_query/dataset" \
  -H "Authorization: ApiKey ${API_KEY}"

DELETE /_query/dataset/{name} deletes a dataset by name.

Deleting a dataset doesn't check whether an ES|QL view reads from it. Views that read from a deleted dataset fail when queried.

DELETE /_query/dataset/access_logs

curl -X DELETE "${ELASTICSEARCH_URL}/_query/dataset/access_logs" \
  -H "Authorization: ApiKey ${API_KEY}"

After creating a dataset, query it with ES|QL. To change how the dataset selects or interprets files, return to define a dataset.