Fix error: All shards failed


Error: all shards failed

The all shards failed error indicates that Elasticsearch couldn't get a successful response from any of the shards involved in the query. Possible causes include shard allocation issues, misconfiguration, insufficient resources, or unsupported operations such as aggregating on text fields.

Detect and resolve issues with AutoOps

AutoOps is a monitoring tool that detects issues in your Elasticsearch cluster in real time and recommends how to resolve them and improve performance. Learn more about AutoOps.

Unsupported operations on text fields

The all shards failed error can occur when you try to sort or aggregate on text fields. These fields are designed for full-text search and don't support exact-value operations like sorting and aggregation.

To fix this issue, use a .keyword subfield:

GET my-index/_search
{
  "aggs": {
    "names": {
      "terms": {
        "field": "name.keyword"
      }
    }
  }
}

If no .keyword subfield exists, define a multi-field mapping:

PUT my-index
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

Metric aggregations on text fields

The all shards failed error can also occur when you use a metric aggregation on a text field. Metric aggregations require numeric fields.

You can use a script to convert the text value to a number at query time:

GET my-index/_search
{
  "aggs": {
    "total_cost": {
      "sum": {
        "script": {
          "source": "Integer.parseInt(doc.cost.value)"
        }
      }
    }
  }
}

Or change the field mapping to a numeric type:

PUT my-index
{
  "mappings": {
    "properties": {
      "cost": {
        "type": "integer"
      }
    }
  }
}

Failed shard recovery

A shard failure during recovery can prevent successful queries.

To identify the cause, check the cluster health:

GET _cluster/health

As a last resort, you can delete the problematic index.

Misused global aggregation

Global aggregations must be defined at the top level of the aggregations object. Nesting can cause errors.

To fix this issue, structure the query so that the global aggregation appears at the top level:

GET my-index/_search
{
  "size": 0,
  "aggs": {
    "all_products": {
      "global": {},
      "aggs": {
        "genres": {
          "terms": {
            "field": "cost"
          }
        }
      }
    }
  }
}

Reverse_nested usage errors

Using a reverse_nested aggregation outside of a nested context causes errors.

To fix this issue, structure the query so that the reverse_nested aggregation is inside a nested aggregation:

GET my-index/_search
{
  "aggs": {
    "comments": {
      "nested": {
        "path": "comments"
      },
      "aggs": {
        "top_usernames": {
          "terms": {
            "field": "comments.username"
          },
          "aggs": {
            "comment_issue": {
              "reverse_nested": {},
              "aggs": {
                "top_tags": {
                  "terms": {
                    "field": "tags"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Further troubleshooting

Use the _cat/shards API to view shard status and troubleshoot further.

GET _cat/shards

For a specific index:

GET _cat/shards/my-index

Example output:

my-index 5 p STARTED    0  283b 127.0.0.1 ziap
my-index 5 r UNASSIGNED
my-index 2 p STARTED    1 3.7kb 127.0.0.1 ziap
my-index 2 r UNASSIGNED
my-index 3 p STARTED    3 7.2kb 127.0.0.1 ziap
my-index 3 r UNASSIGNED
my-index 1 p STARTED    1 3.7kb 127.0.0.1 ziap
my-index 1 r UNASSIGNED
my-index 4 p STARTED    2 3.8kb 127.0.0.1 ziap
my-index 4 r UNASSIGNED
my-index 0 p STARTED    0  283b 127.0.0.1 ziap
my-index 0 r UNASSIGNED