Disk watermark flood stage

Disk usage on one or more non-frozen nodes has crossed the flood-stage watermark. Elasticsearch blocks writes to all indices with shards on those nodes; reads still work.

Note

For a complete list of insights, refer to AutoOps insights.

Insight details

Field	Value
Component	Elasticsearch
Severity	Critical
Scope	Node
Domains	cluster-health, disk-storage, stability

Customization settings

You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.

The default customization settings are:

Setting	Type	Default
Early warning margin before the flood-stage watermark	Percentage	`0`

Example: What you might see in AutoOps

The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.

The flood-stage disk watermark has been exceeded on: es-data-01 (hot, warm)

What was detected

The flood-stage disk watermark has been exceeded by 12 GB on non-frozen nodes.

Affected nodes: es-data-01 and es-data-02.

Affected tier(s): hot, warm.

Remaining space before full capacity: 48 GB.

Largest indices on the affected node(s): logs-prod-000045.

Recommendations

Note

AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.

Increase disk size

Condition: Shown when flood-stage watermark is crossed on a non-frozen data node.

Increase disk size on the affected node (current capacity: 500 GB).

Temporary disk watermark increase

Condition: Shown when available disk space > 100GB.

Temporarily raise the disk watermark so you can free space before the cluster restricts shard allocation. After you reclaim disk, return the watermark to its previous value. Use the action below to update cluster.routing.allocation.disk.watermark.high.

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.high": "90%"
  }
}

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Move shard to less loaded node

Condition: Shown when other non-frozen data nodes have spare space.

Move shard 0 of logs-prod-000045 from es-data-01 to es-data-02 with the action below to rebalance disk use.

POST _cluster/reroute
{
  "commands": [
    {
      "move": {
        "index": "logs-prod-000045",
        "shard": 0,
        "from_node": "es-data-01",
        "to_node": "es-data-02"
      }
    }
  ]
}

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Reduce shard replicas

Condition: Shown when number of replicas > 1.

Reduce replica count on logs-prod-000045 from 2 to 1 to free disk space. Start with indices that have more than two shard copies.

PUT logs-prod-000045/_settings
{
  "index": {
    "number_of_replicas": 1
  }
}

Note

Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Delete old data

Condition: Shown when "old" index exists; warm/cold tier: consider retention policies and ILM.

Delete unused or old indices to free disk space. Snapshot indices first if you might need them later. Use the action below to remove logs-archive-000001.

DELETE logs-archive-000001

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Force merge deleted documents

Condition: Shown when old read-only indices exist on affected non-frozen data node (particularly warm/cold tier).

Run a force merge with only_expunge_deletes=true on logs-prod-000045, logs-prod-000046 to reclaim disk used by deleted documents. Use the action below.

POST logs-archive-000001/_forcemerge?only_expunge_deletes=true

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Background and impact

The flood-stage watermark is the final disk pressure threshold for non-frozen data nodes. When it is crossed, Elasticsearch sets all indices with at least one shard on the affected node to read-only. Searches still work, but writes and updates are blocked until disk space is freed and the read-only flag is lifted.

Respond immediately. On hot nodes, write blocking means active ingestion is interrupted. On warm and cold nodes, the primary concern is freeing space through ILM, data deletion, or shard relocation. After you recover space, you might need to manually clear the index read-only setting (index.blocks.write) for affected indices.

Guide

Watermark errors