Set up a time series data stream

Create an index lifecycle policy (optional)

In most cases, you can use a data stream lifecycle to manage your time series data stream. If you're using data tiers in Elastic Stack, you can use index lifecycle management.

PUT _ilm/policy/my-weather-sensor-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          }
        }
      }
      // Additional phases (warm, cold, delete) as needed
    }
  }
}

Create an index template

The structure of a time series data stream is defined by an index template. Create an index template with the following required elements and settings:

• Index patterns: One or more wildcard patterns matching the name of your TSDS, such as weather-sensors-*. For best results, use the data stream naming scheme.
• Data stream object: The template must include "data_stream": {}.
• Time series mode: Set index.mode: time_series.
• Field mappings: Define at least one dimension field and typically one or more metric fields:
  • Dimensions: To define a dimension, set time_series_dimension to true. For details, refer to Dimensions.
    • To define dimensions dynamically, you can use a pass-through object. For details, refer to Defining sub-fields as time series dimensions.
  • Metrics: To define a metric, use the time_series_metric mapping parameter. For details, refer to Metrics.
  • Timestamp (optional): Define a date or date_nanos mapping for the @timestamp field. If you don't specify a mapping, Elasticsearch maps @timestamp as a date field with default options.
  • Serverless Unavailable Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Unavailable This functionality is not available in Elastic Cloud Serverless. Stack The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Lifecycle management: If you're using Elastic Stack, define a lifecycle to enable automatic rollover and prevent indices from growing too large.
    • Add a lifecycle object and specify "enabled": true.
    • If you created an ILM policy in step 1, reference it in the settings with index.lifecycle.name.
  • Settings (optional): Define any relevant index settings, such as index.number_of_replicas, for the data stream's backing indices.
• Priority: Set the priority higher than 200 to avoid collisions with built-in templates.

Example index template PUT request:

PUT _index_template/my-weather-sensor-index-template
{
  "index_patterns": ["metrics-weather_sensors-*"],
  "data_stream": { },
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.lifecycle.name": "my-lifecycle-policy"
    },
    "lifecycle": { 
      "enabled": true
    },
    "mappings": {
      "properties": {
        "sensor_id": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "location": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "temperature": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "humidity": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "@timestamp": {
          "type": "date"
        }
      }
    }
  },
  "priority": 500,
  "_meta": {
    "description": "Template for my weather sensor data"
  }
}

1. Elastic Stack only
2. Elastic Stack only, enabled by default in Serverless

Turn on past index creation (optional)

Elasticsearch can create missing backing indices when you add data that precedes existing time ranges. To enable this feature, update the cluster settings:

PUT _cluster/settings
{
  "persistent": {
    "data_stream.past_tsdb_index_creation_enabled": true,
    "data_streams.past_tsdb_index_interval": "2d"
  }
}

1. These settings are persistent and apply to all data streams in the cluster.
2. By default, each past backing index covers one day of data. Refer to data_streams.past_tsdb_index_interval.

Create the time series data stream and add data

You can create a time series data stream explicitly with the create a data stream API. You must give it a name that matches the index_patterns in your index template. For example:

PUT _data_stream/metrics-weather-sensors

Alternatively, you can create it implicitly by indexing a document. The TSDS is created automatically as long as the index name in your API matches the index template pattern.

PUT metrics-weather-sensors/_bulk
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "SENSOR-001", "location": "warehouse-A", "temperature": 26.7,"humidity": 49.9 }
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SENSOR-002", "location": "warehouse-B", "temperature": 32.4, "humidity": 88.9 }
POST metrics-weather-sensors/_doc
{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "sensor_id": "SENSOR-00002",
  "location": "warehouse-B",
  "temperature": 32.4,
  "humidity": 88.9
}

For more information about adding documents to data streams, go to Use a data stream > Add documents to a data stream.

Verify setup

To make sure your time series data stream is working, try some GET requests.

View data stream details:

GET _data_stream/metrics-weather-sensors

Check the document count in a time series data stream:

GET metrics-weather-sensors/_count 

Query the time series data:

GET metrics-weather-sensors/_search 
{
  "size": 5,
  "sort": ["@timestamp"]
}