Oversized shards

One or more shards exceed the recommended size range (roughly 50 GB, or a very high document count). Oversized shards slow recovery, relocation, and search, and in extreme cases can become hard to move or write.

Note

For a complete list of insights, refer to AutoOps insights.

Insight details

Field	Value
Component	Elasticsearch
Severity	Medium
Scope	Index
Domains	performance, index-management, shard-allocation

Customization settings

You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.

The default customization settings are:

Setting	Type	Default
Maximum optimal shard size in GB	Integer	`0.0004`
Exclude indices based on this pattern	List of strings

Example: What you might see in AutoOps

The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.

There are large shards in the cluster

What was detected

Total number of indices with large shards: 12

The index that contains the largest shard is: logs-prod-000045, and the shard size is: 42 GB.

See a sample list: logs-prod-000045, logs-prod-000046.

Recommendations

Note

AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.

ILM policy for large shards

Condition: Always shown for this insight.

If logs-prod-000045 is time-based, set or tune its ILM policy so rollover targets roughly 20–30 GB shards.

Reindex into smaller indices

Condition: Shown for non-system indexes.

Reindex into smaller indices to reduce shard sizes and improve search performance.

Split index into more shards

Condition: Shown when primary shard count is less than data node count and shard size warrants split.

Use the Split API to split logs-prod-000045 (current size 42 GB) into 2 primary shards with the action below. Run during low cluster load and confirm you have enough disk space. See the Split index API for prerequisites.

POST logs-prod-000045/_split/logs-prod-000045-0001
{
  "settings": {
    "index.number_of_shards": 2
  }
}

Note

Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.

Background and impact

Index data is partitioned into shards when the index is created. Shard size grows with the data you write into that index.

Shards that grow well beyond about 50 GB, or that hold hundreds of millions of documents, take longer to recover and relocate. Searches that hit those shards can also run more slowly. In extreme cases a shard can approach hard limits (for example around 2 billion documents) and become difficult to write to or move.

Keeping shards in a moderate size range helps the cluster recover faster and keeps search and indexing more predictable as data volume grows.

Guide

Size your shards