Status yellow

Cluster health is yellow because one or more replica shards are not allocated, while primaries are generally serving reads and writes. You still have service, but losing another node could leave indices without redundancy.

Note

For a complete list of insights, refer to AutoOps insights.

Insight details

Field	Value
Component	Elasticsearch
Severity	Medium
Scope	Cluster
Domains	cluster-health, high-availability

Customization settings

You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.

The default customization settings are:

Setting	Type	Default
Minimum number of initializing shards	Integer	`1`
Minimum duration of the yellow status in seconds	Integer	`60`
Reasons excluded to raise a yellow status	List of strings	`REPLICA_ADDED,DANGLING_INDEX_IMPORTED,CLUSTER_RECOVERED,INDEX_CLOSED,INDEX_REOPENED,EXISTING_INDEX_RESTORED,NEW_INDEX_RESTORED,REALLOCATED_REPLICA,MANUAL_ALLOCATION`
Minimum number of data nodes required to raise the yellow status event	Integer	`2`

Example: What you might see in AutoOps

The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.

The cluster status is yellow

What was detected

The current number of unassigned replica shards is: 12.

The maximum number unassigned replica shards is: 12.

The number of initializing shards is: 12.

The shards are unassigned because:

logs-prod-000045: NODE_LEFT

See sample list: logs-prod-000045, logs-prod-000046.

Recommendations

Note

AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.

Wait for shard recovery to complete

Condition: Shown when only initializing shards.

Shard recovery is in progress after data nodes restarted or were removed: es-data-01. Wait until recovery finishes before making other cluster changes.

Reduce shard replicas

Condition: Shown when unassigned shards and number of replicas > 1.

Reduce replica count on logs-prod-000045 from 2 to 1 to free disk space. Start with indices that have more than two shard copies.

PUT logs-prod-000045/_settings
{
  "index": {
    "number_of_replicas": 1
  }
}

Note

Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Add data node

Condition: Shown when total replicas / total primaries > number of data nodes.

Add a data node to increase capacity and reduce pressure on the existing nodes.

Retry failed shard allocations

Condition: Shown when any shard failed to be allocated.

If shards failed to allocate and nothing is recovering, retry allocation with a cluster reroute. Use the action below (retry_failed=true).

POST _cluster/reroute?retry_failed=true

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Explore shard allocation in the cluster

Condition: Shown when no initializing shards.

Use cluster allocation explain to see why shards are unassigned for index logs-prod-000045, shard 0. Run the action below.

GET _cluster/allocation/explain
{
  "index": "logs-prod-000045",
  "shard": 0,
  "primary": true
}

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action is read-only.

Check total shards per node limit

Condition: Shown when no initializing shards.

index.routing.allocation.total_shards_per_node on logs-prod-000045 might block replica assignment. Fetch index settings with the action below and adjust or remove the limit if it causes unassigned shards.

GET logs-prod-000045/_settings?include_defaults=true&filter_path=**.index.routing.allocation.total_shards_per_node

Note

Requires the manage index privilege. Requires Elasticsearch 8.0.0 or later. This action is read-only.

Background and impact

Impact: When cluster status is yellow, it means there is a higher risk of permanent or temporary loss of data.

Yellow status indicates that one or more of the replica shards on the cluster are not allocated to a node.

While the cluster status is yellow, all data is still accessible for read and write operations, but in the event of data corruption or loss of a node, one or more of your indices will no longer be available for write or search.

If you only have one node and your index has replica shards, then a yellow status is normal, since no other node is available to receive the replica.

A cluster might also become yellow temporarily following a node restart, until the node resynchronizes the replica shards. However, if the yellow status persists, you should act to find out why and recover your cluster to Green status.

If the shard recovery process has been completed and the cluster status is still yellow, it's an indication that the allocation was not carried out successfully. There could be various reasons for this failure:

There is only 1 node in your cluster, and replicas will never be allocated on the same node as the primary shard.
One or more nodes have disk utilization that is above the cluster.routing.allocation.disk.watermark.low value.
The use of attribute-based allocation rules is preventing a shard from being allocated on any of the available nodes.