Configuring a time series data stream for downsampling

To downsample a time series using a data stream lifecycle, add a downsampling section to the data stream lifecycle (for existing data streams) or the index template (for new data streams).

• Set fixed_interval to your preferred level of granularity. The original time series data will be aggregated at this interval.
• Set after to the minimum time to wait after an index rollover, before running downsampling.
• Stack Preview 9.3+ The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Preview since 9.3 This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Serverless Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Generally available This functionality is generally available and ready for production usage. Serverless interfaces and procedures might differ from classic Elastic Stack deployments. (Optional) Set downsampling_method to your preferred downsampling method, or leave it unspecified to use the default method (aggregate).

PUT _data_stream/my-data-stream/_lifecycle
{
  "data_retention": "7d",
  "downsampling_method": "aggregate",
  "downsampling": [
     {
       "after": "1m",
       "fixed_interval": "10m"
      },
      {
        "after": "1d",
        "fixed_interval": "1h"
      }
   ]
}

The downsampling action runs after the index time series end time has passed.

Stack 9.5+ The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available since 9.5 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Serverless Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Generally available This functionality is generally available and ready for production usage. Serverless interfaces and procedures might differ from classic Elastic Stack deployments. Data stream lifecycle limits how many backing indices per data stream it submits for downsampling at once. When the number of indices actively being downsampled reaches the threshold configured by data_streams.lifecycle.downsampling.max_indices_in_progress, data stream lifecycle pauses queuing further downsampling operations for that data stream until the count drops. Deferred indices are recorded in the error store with a warning and retried in a subsequent lifecycle run. Other data streams are not affected.

This limit helps prevent cluster overload when many indices become eligible for downsampling at the same time, such as after loading historical metrics and enabling lifecycle on the data stream.

To downsample time series data as part of index lifecycle management (ILM), include downsample actions in your ILM policy. You can configure multiple downsampling actions across different phases to progressively reduce data granularity over time.

This example shows a policy with rollover and two downsampling actions: one in the hot phase for initial aggregation at 5-minute intervals, and another in the warm phase for further aggregation at 1-hour intervals:

PUT _ilm/policy/datastream_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_age": "5m"
          },
          "downsample": {
  	        "fixed_interval": "5m",
  	        "sampling_method": "aggregate"
  	      }
        }
      },
      "warm": {
        "actions": {
          "downsample": {
            "fixed_interval": "1h",
  	        "sampling_method": "aggregate"
          }
        }
      }
    }
  }
}

• Set fixed_interval to your preferred level of granularity. The original time series data will be aggregated at this interval. The downsample action runs after the index is rolled over and the index time series end time has passed.
• Stack Preview 9.3+ The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Preview since 9.3 This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. (Optional) Set sampling_method to your preferred downsampling method, or leave it unspecified to use the default method (aggregate).