Disk watermark high

Disk usage on one or more nodes has crossed the high watermark. Elasticsearch begins relocating shards away from those nodes.

Note

For a complete list of insights, refer to AutoOps insights.

Insight details

Field	Value
Component	Elasticsearch
Severity	High
Scope	Node
Domains	cluster-health, disk-storage, stability

Customization settings

You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.

The default customization settings are:

Setting	Type	Default
Early warning margin before the high watermark	Percentage	`0`

Example: What you might see in AutoOps

The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.

The high disk watermark has been exceeded on: es-data-01 (hot, warm)

What was detected

The high disk watermark has been exceeded by 12 GB.

Affected nodes: es-data-01 and es-data-02.

Affected tier(s): hot, warm.

Remaining space before the flood-stage watermark: 48 GB.

Largest indices on the affected node(s): logs-prod-000045.

Recommendations

Note

AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.

Increase disk size

Condition: Shown when disk watermark is crossed on a data node; on hot nodes this is ingest risk, on warm/cold nodes this is retention risk.

Increase disk size on the affected node (current capacity: 500 GB).

Temporary disk watermark increase

Condition: Shown when available disk space > 100GB.

Temporarily raise the disk watermark so you can free space before the cluster restricts shard allocation. After you reclaim disk, return the watermark to its previous value. Use the action below to update cluster.routing.allocation.disk.watermark.high.

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.high": "90%"
  }
}

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Move shard to less loaded node

Condition: Shown when other data nodes (preferably in the same tier) have spare space.

Move shard 0 of logs-prod-000045 from es-data-01 to es-data-02 with the action below to rebalance disk use.

POST _cluster/reroute
{
  "commands": [
    {
      "move": {
        "index": "logs-prod-000045",
        "shard": 0,
        "from_node": "es-data-01",
        "to_node": "es-data-02"
      }
    }
  ]
}

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Reduce shard replicas

Condition: Shown when number of replicas > 1.

Reduce replica count on logs-prod-000045 from 2 to 1 to free disk space. Start with indices that have more than two shard copies.

PUT logs-prod-000045/_settings
{
  "index": {
    "number_of_replicas": 1
  }
}

Note

Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Delete old data

Condition: Shown when "old" index exists; warm/cold tier: consider retention policies and ILM.

Delete unused or old indices to free disk space. Snapshot indices first if you might need them later. Use the action below to remove logs-archive-000001.

DELETE logs-archive-000001

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Force merge deleted documents

Condition: Shown when old read-only indices exist on affected data node (particularly warm/cold tier).

Run a force merge with only_expunge_deletes=true on logs-prod-000045, logs-prod-000046 to reclaim disk used by deleted documents. Use the action below.

POST logs-archive-000001/_forcemerge?only_expunge_deletes=true

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Background and impact

When the high watermark is crossed, Elasticsearch actively relocates shards away from the affected nodes to relieve disk pressure. Writes to existing shards on those nodes continue until the flood-stage watermark is reached, at which point indices are made read-only.

Act promptly. On hot nodes, high disk usage directly threatens ingest throughput. On warm or cold nodes, ILM transitions or data deletion are typically the right levers. On any tier, if relocation is not enough, expand disk capacity or apply index retention policies before the flood stage is reached.

Guide

Watermark errors