Shard count high on frozen node
Elastic Cloud Hosted lets you manage and configure one or more deployments of the versioned Elastic Stack, hosted on Elastic Cloud. Availability This functionality is generally available and ready for production usage. Elastic Cloud on Kubernetes extends Kubernetes orchestration capabilities to allow you to deploy and manage components of the Elastic Stack. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Elastic Cloud Enterprise is a self-managed orchestration platform for deploying and managing the Elastic Stack at scale. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Self-managed deployments are Elastic Stack deployments managed without the assistance of an orchestrator. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.
ECH
Generally available
ECK
Generally available
ECE
Generally available
Self-managed
Generally available
One or more frozen tier nodes host more shards than the recommended limit for stable operation. High shard counts can cause master nodes to struggle with cluster state tracking and can stress file descriptor and virtual memory limits on the affected nodes.
For a complete list of insights, refer to AutoOps insights.
| Field | Value |
|---|---|
| Component | Elasticsearch |
| Severity | Medium |
| Scope | Node |
| Domains | shard-allocation, stability |
You can customize these settings to adjust when AutoOps detects this event and presents the insight. Refer to AutoOps event settings for details.
The default customization settings are:
| Setting | Type | Default |
|---|---|---|
| Early warning margin before the max shards per frozen node limit | Percentage | 10 |
The following is an example of what you might see when this insight is triggered. Real insights use live data and links from your deployment or cluster.
The current shard count on es-data-01 and es-data-02 is 2700 (limit: 3000).
AutoOps shows different recommendations depending on how their conditions match your deployment or cluster.
Add frozen data node
Condition: Shown when frozen node shard count is at or near limit.
Add a frozen-tier data node to host searchable snapshot indices and scale frozen storage separately.
Increase the maximum shards per node limit
Condition: Shown when frozen node is at or near the max_shards_per_node.frozen limit; recommend raising the limit by about 10% (whole number, round up).
Your cluster is approaching the configured maximum shards per node (currently 3000 for cluster.max_shards_per_node.frozen). You can raise this limit to 3300 to unblock shard allocation temporarily, or increase it permanently after a cluster sizing review. Before increasing the limit, confirm that your nodes have sufficient memory and file descriptors to support additional shards. Run the following command to update the setting.
PUT _cluster/settings?pretty{ "persistent": { "cluster.max_shards_per_node.frozen": "3300" } }
Requires the manage cluster privilege. Requires Elasticsearch 8.0.0 or later. This action changes cluster or index configuration.
Your cluster has more shards on one or more frozen nodes than is recommended for stable operation. Oversharding can destabilize a cluster when master nodes struggle to track many shards across nodes.
High shard counts can also approach limits for virtual memory (mmap) and open file descriptors on the affected nodes. Both might need adjustment if shard counts increase further.
In the version (8.15.0) you're running, the memory usage per shard is significantly higher than in more recent versions, where a pivotal change was introduced to optimize memory consumption. Consider upgrading to a more recent version. If that is not an option right now, lower the cluster.max_shards_per_node.frozen cluster setting to keep memory consumption in check.