Shard count high on data node
Elastic Cloud Hosted lets you manage and configure one or more deployments of the versioned Elastic Stack, hosted on Elastic Cloud. Availability This functionality is generally available and ready for production usage. Elastic Cloud on Kubernetes extends Kubernetes orchestration capabilities to allow you to deploy and manage components of the Elastic Stack. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Elastic Cloud Enterprise is a self-managed orchestration platform for deploying and managing the Elastic Stack at scale. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Self-managed deployments are Elastic Stack deployments managed without the assistance of an orchestrator. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.
ECH
Generally available
ECK
Generally available
ECE
Generally available
Self-managed
Generally available
One or more data nodes host more shards than the recommended limit. Oversharding strains master coordination and increases per-shard memory use, especially on older Elasticsearch versions.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | Medium |
| Scope | Node |
| Domains | shard-allocation, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Early warning margin before the max shards per node limit | Percentage | 30 |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The current number of shards on es-data-01 and es-data-02 is 750 (limit: 1000).
- Index logs-prod-000045 has 12 total primary shards, of which 12 are on this node.
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Increase the maximum shards per node limit
Condition: Shown when node is at or near the max_shards_per_node limit; recommend raising the limit by about 10% (whole number, round up).
Your cluster is approaching the configured maximum shards per node (currently 1000 for cluster.max_shards_per_node). You can raise this limit to 1100 to unblock shard allocation temporarily, or increase it permanently after a cluster sizing review. Before increasing the limit, confirm that your nodes have sufficient memory and file descriptors to support additional shards. Run the following command to update the setting.
PUT _cluster/settings?pretty{ "persistent": { "cluster.max_shards_per_node": "1100" } }
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Merge similar indices
Condition: Always shown for this insight.
Merge similar indices to reduce the number of small indices and improve cluster performance and stability.
Add data node
Condition: Always shown for this insight.
Add a data node to increase capacity and reduce pressure on the existing nodes.
Too many shards on a data node can destabilize the cluster. Master nodes must track every shard in the cluster state, and a high shard count on individual nodes increases coordination overhead and can slow down cluster state updates.
High shard counts can also approach limits for virtual memory (mmap) and open file descriptors on the affected nodes. Both might need adjustment if shard counts increase further.
In the version (8.15.0) you're running, the memory usage per shard is significantly higher than in more recent versions, where a pivotal change was introduced to optimize memory consumption. Consider upgrading to a more recent version. If that is not an option right now, lower the cluster.max_shards_per_node cluster setting to keep memory consumption in check.