Shard near document limit

One or more shards are approaching the practical maximum document count per shard (about 200 million documents). Uneven growth can come from parent/child or nested routing, too few primary shards, or missing rollover for time-series indices.

Note

For a complete list of insights, refer to AutoOps insights.

Insight details

Field	Value
Component	Elasticsearch
Severity	High
Scope	Index
Domains	disk-storage, index-management

Customization settings

You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.

The default customization settings are:

Setting	Type	Default
Exclude indices based on this pattern	List of strings

Example: What you might see in AutoOps

The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.

One or more shards are approaching the max document limit

What was detected

More than 25 indices have shards approaching the maximum document count.

The shard with the most documents is on logs-prod-000045 (42 GB documents).

Other affected indices: logs-prod-000045, logs-prod-000046.

Recommendations

Note

AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.

Split index into more shards

Condition: Shown when primary shard count is less than data node count.

Use the Split API to split logs-prod-000045 (current size 42 GB) into 2 primary shards with the action below. Run during low cluster load and confirm you have enough disk space. See the Split index API for prerequisites.

POST logs-prod-000045/_split/logs-prod-000045-0001
{
  "settings": {
    "index.number_of_shards": 2
  }
}

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Rollover indices

Condition: Shown for time-series indices approaching document limit.

If logs-prod-000045 are time-based, roll them over and set 1 primary shards on the new write index so you can avoid a heavy segment merge on the current indices.

Reindex into smaller indices

Condition: Shown when primary shard count is less than data node count.

Reindex into smaller indices to reduce shard sizes and improve search performance.

Split index into more shards

Condition: Shown when shard document count approaches limit and split is viable.

Use the Split API to split logs-prod-000045 into a new index with more primary shards. Its largest shard currently holds 42 GB documents.

Background and impact

Shards in 12 are approaching the maximum recommended document count for a single shard (200,000,000).

This can happen when parent/child or nested documents route child docs to the same shard as the parent. It can also happen when an index has too few primary shards, or when time-series indices are not rolled over before they grow too large.

If a shard reaches the limit, indexing to that shard can fail until you split, reindex, or roll over.

Guide

Size your shards