Slow search on warm tier nodes

Search on warm-tier nodes exceeded the warm-tier latency threshold for consecutive samples. Warm data sits between hot and cold in cost and access pattern, but slow warm search still delays reporting and tiered analytics.

Note

For a complete list of insights, refer to AutoOps insights.

Insight details

Field	Value
Component	Elasticsearch
Severity	High
Scope	Node
Domains	performance, search

Customization settings

You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.

The default customization settings are:

Setting	Type	Default
Search latency threshold (ms)	Integer	`2000`
Consecutive samples above threshold	Integer	`5`

Tip

Raising these thresholds reduces noise but delays detection. Lowering them triggers the insight sooner but can increase alerts during minor blips.

Example: What you might see in AutoOps

The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.

Slow search detected on es-data-01

What was detected

Search latency on es-data-01 and es-data-02 on warm-tier nodes stayed above your configured threshold for enough consecutive samples. Peak latency in the latest sample was 420 ms.

Indices with high search activity: logs-prod-000045

Review query logs or search slow logs on the affected node, tune expensive queries, and check CPU and heap load on the same node if latency stays high.

Recommendations

Note

AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.

Increase replica count

Condition: Shown when index has no replica.

Set number_of_replicas to 1 on logs-prod-000045 (currently 2) using the action below.

PUT logs-prod-000045/_settings
{
  "index": {
    "number_of_replicas": 1
  }
}

Note

Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Review query logs

Condition: Shown when cluster version is 9.4 or above.

Review query logs on es-data-01 to find expensive searches that fill the queue. Query logs (Elasticsearch 9.4+) give structured per-query timing with less overhead than search slow logs. See Query logs.

Enable and review search slow logs

Condition: Shown when cluster version is below 9.4.

Enable search slow logs with the action below, then review the slow log to find expensive queries. See Slow logs for configuration details.

PUT logs-prod-000045/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "5s",
  "index.search.slowlog.threshold.query.debug": "2s",
  "index.search.slowlog.threshold.query.trace": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.search.slowlog.threshold.fetch.info": "800ms",
  "index.search.slowlog.threshold.fetch.debug": "500ms",
  "index.search.slowlog.threshold.fetch.trace": "200ms"
}

Note

Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Add data node

Condition: Shown when index has more than twice as many pri/rep shards as available data nodes.

Add a data node to increase capacity and reduce pressure on the existing nodes.

Background and impact

When search stays slow on a warm-tier node, dashboards and analytics jobs that rely on warm data feel sluggish. Common causes include expensive or repeated queries, CPU or heap pressure, and background work such as segment merges or snapshots.

If latency remains high, focus on the indices in the preceding section. Add replicas or data capacity where shard layout limits parallelism. Your log threshold settings control which queries are recorded — tune them to capture the queries most likely driving the latency.