Enabling and disabling advanced indexing features for Fleet-managed data streams

Fleet provides support for several advanced features around its data streams, including:

• Time series data streams (TSDS)
• Synthetic _source

These features can be enabled and disabled for Fleet-managed data streams by using the index template API and a few key settings. In versions 8.17.0 and later, synthetic _source is available only for certain Elastic subscription levels.

We recommended using Kibana Dev Tools to run the following requests. Replace <NAME> with the name of a given integration data stream. For example specifying metrics-nginx.stubstatus results in making a PUT request to _component_template/metrics-nginx.stubstatus@custom. Use the index management interface to explore what integration data streams are available to you.

Once you’ve executed a given request below, you also need to execute a data stream rollover to ensure any incoming data is ingested with your new settings immediately. For example:

POST metrics-nginx.stubstatus-default/_rollover

Refer to the following steps to enable or disable advanced data stream features:

• Disable synthetic _source

Due to restrictions in the Elasticsearch API, TSDS must be enabled at the index template level. So, you’ll need to make some sequential requests to enable or disable TSDS.

1. Send a GET request to retrieve the index template:

   GET _index_template/<NAME>
   

2. Use the JSON payload returned from the GET request to populate a PUT request, for example:

   PUT _index_template/<NAME>
   {
     # You can copy & paste this directly from the GET request above
     "index_patterns": [
       "<index pattern from GET request>"
     ],
   
     # Make sure this is added
     "template": {
       "settings": {
         "index": {
           "mode": "time_series"
         }
       }
     },
   
     # You can copy & paste this directly from the GET request above
     "composed_of": [
       "<NAME>@package",
       "<NAME>@custom",
       ".fleet_globals-1",
       ".fleet_agent_id_verification-1"
     ],
   
     # You can copy & paste this directly from the GET request above
     "priority": 200,
   
     # Make sure this is added
     "data_stream": {
       "allow_custom_routing": false
     }
   }
   

To disable TSDS, follow the same procedure as to enable TSDS, but specify null for index.mode instead of time_series. Follow the steps below or you can copy the NGINX example.

1. Send a GET request to retrieve the index template:

   GET _index_template/<NAME>
   

2. Use the JSON payload returned from the GET request to populate a PUT request, for example:

   PUT _index_template/<NAME>
   {
     # You can copy/paste this directly from the GET request above
     "index_patterns": [
       "<index pattern from GET request>"
     ],
   
     # Make sure this is added
     "template": {
       "settings": {
         "index": {
           "mode": null
         }
       }
     },
   
     # You can copy/paste this directly from the GET request above
     "composed_of": [
       "<NAME>@package",
       "<NAME>@custom",
       ".fleet_globals-1",
       ".fleet_agent_id_verification-1"
     ],
   
     # You can copy/paste this directly from the GET request above
     "priority": 200,
   
     # Make sure this is added
     "data_stream": {
       "allow_custom_routing": false
     }
   }
   

   For example, the following payload disables TSDS on nginx.stubstatus:

   {
     "index_patterns": [
         "metrics-nginx.stubstatus-*"
     ],
   
     "template": {
       "settings": {
         "index": {
           "mode": null
         }
       }
     },
   
     "composed_of": [
       "metrics-nginx.stubstatus@package",
       "metrics-nginx.stubstatus@custom",
       ".fleet_globals-1",
       ".fleet_agent_id_verification-1"
     ],
   
     "priority": 200,
   
     "data_stream": {
       "allow_custom_routing": false
     }
   }
   

PUT _component_template/<NAME>@custom
{
  "settings": {
    "index": {
      "mapping": {
        "source": {
          "mode": "synthetic"
        }
      }
    }
  }
}

PUT _component_template/<NAME>@custom
{
  "settings": {
    "index": {
      "mapping": {
        "source": {"mode": "stored"}
      }
    }
  }
}