Data stream lifecycle in Elasticsearch

A data stream lifecycle in Elasticsearch is a built-in automation mechanism for managing data retention, performance, and storage optimization. By configuring rollover, retention, downsampling, and frozen transitions to long-term searchable archives, your data streams stay efficient as they age. This page explains how the lifecycle works, its key features, and how to configure it for both new and existing data streams.

Data stream lifecycle manages your data streams according to your retention and storage goals. For example, you can configure the lifecycle to:

• Ensure that data indexed in the data stream is kept at least for the retention time you defined.
• Ensure that data older than the retention period is deleted automatically by Elasticsearch at a later time.
• Stack 9.5+ The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available since 9.5 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Serverless Unavailable Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Unavailable This functionality is not available in Elastic Cloud Serverless. Move older backing indices to low-cost searchable storage. Refer to Searchable snapshots for data streams.

To achieve these, data stream lifecycle supports:

• Automatic rollover, which chunks your incoming data in smaller pieces to facilitate better performance and backwards incompatible mapping changes.
• Configurable retention, which allows you to configure the time period for which your data is guaranteed to be stored. Elasticsearch is allowed at a later time to delete data older than this time period. Retention can be configured on the data stream level or on a global level. Read more about the different options in this tutorial.

Data stream lifecycle also supports downsampling the data stream backing indices. Refer to the downsampling example for more details.

Note the availability of data stream lifecycle to ensure that it's applicable for your use case:

• Data stream lifecycle is supported only for data streams and cannot be used with individual indices.

• Data stream lifecycle is supported for all deployment types on the versioned Elastic Stack as well as for Elasticsearch Serverless.

In intervals configured by data_streams.lifecycle.poll_interval, Elasticsearch goes over each data stream and performs the following steps:

1. Checks if the data stream has a data stream lifecycle configured, skipping any indices not part of a managed data stream.
2. Rolls over the write index of the data stream, if it fulfills the conditions defined by cluster.lifecycle.default.rollover.
3. After an index is not the write index anymore (that is, the data stream has been rolled over), automatically tail merges the index. Data stream lifecycle runs a merge operation that only targets the long tail of small segments instead of the whole shard. As the segments are organised into tiers of exponential sizes, merging the long tail of small segments is only a fraction of the cost of force merging to a single segment. The small segments would usually hold the most recent data so tail merging will focus the merging resources on the higher-value data that is most likely to keep being queried.
4. If downsampling is configured it will run all the configured downsampling rounds. Stack 9.5+ The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available since 9.5 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Serverless Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Generally available This functionality is generally available and ready for production usage. Serverless interfaces and procedures might differ from classic Elastic Stack deployments. Data stream lifecycle limits concurrent downsampling per data stream using the data_streams.lifecycle.downsampling.max_indices_in_progress cluster setting (default 10). For details, refer to Downsample with a data stream lifecycle.
5. Stack 9.5+ The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available since 9.5 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Serverless Unavailable Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Unavailable This functionality is not available in Elastic Cloud Serverless. Transitions eligible backing indices to partially mounted searchable snapshots. Refer to Searchable snapshots for data streams.
6. Applies retention to the remaining backing indices. This means deleting the backing indices whose generation_time is longer than the effective retention period (read more about the effective retention calculation). The generation_time is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the index.lifecycle.origination_date setting. Retention can delete indices even after they have been moved to searchable snapshots.

Since the lifecycle is configured on the data stream level, the process to configure a lifecycle on a new data stream and on an existing one differ.

The following tutorials help you set up and manage data streams with data stream lifecycle:

• To create a new data stream with a lifecycle, add the data stream lifecycle as part of the index template that matches the name of your data stream. Refer to Create a data stream with a lifecycle for the detailed steps. When a write operation with the name of your data stream reaches Elasticsearch then the data stream is created with the respective data stream lifecycle.
• To update the lifecycle settings for an individual, existing data stream, use the data stream lifecycle APIs. Refer to Update the lifecycle of a data stream for details.
• Retention settings for data streams can be configured both individually, at the data stream level, and globally, for all data streams in a cluster. To learn more, refer to Setting retention for Elasticsearch data streams.
• To migrate an existing ILM managed data stream to data stream lifecycle, follow the steps in Migrating ILM-managed data streams to data stream lifecycle.
• Stack 9.5+ The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available since 9.5 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Serverless Unavailable Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Unavailable This functionality is not available in Elastic Cloud Serverless. To move older backing indices to searchable snapshots, refer to Searchable snapshots for data streams.