Merge settings

A shard in Elasticsearch is a Lucene index, and a Lucene index is broken down into segments. Segments are internal storage elements in the index where the index data is stored, and are immutable. Smaller segments are periodically merged into larger segments to keep the index size at bay and to expunge deletes.

The merge process uses auto-throttling to balance the use of hardware resources between merging and other activities like search.

The merge scheduler controls the execution of merge operations when they are needed. Merges run on the dedicated merge thread pool. Smaller merges are prioritized over larger ones, across all shards on the node. Merges are disk IO throttled so that bursts, while merging activity is otherwise low, are smoothed out in order to not impact indexing throughput. There is no limit on the number of merges that can be enqueued for execution on the thread pool. However, beyond a certain per-shard limit, after merging is completely disk IO un-throttled, indexing for the shard will itself be throttled until merging catches up.

The merge scheduler supports the following dynamic index settings:

index.merge.scheduler.max_thread_count

The maximum number of threads on a single shard that may be merging at once. Defaults to half the number of processors the JVM sees on the node, with a minimum of 1, which works well for a good solid-state-disk (SSD). If your index is on spinning platter drives instead, decrease this to 1. This default is independent of node.processors. With include_defaults, the reported default is the unclamped value computed on the responding node; it may differ from the value a data node actually uses after clamping to max_merge_count, and from data nodes with a different processor count.

If max_thread_count would exceed index.merge.scheduler.max_merge_count, Elasticsearch reduces it to max_merge_count when the index is opened or its settings change, and logs a warning. That can happen when max_thread_count is left unset and max_merge_count is set explicitly, especially across nodes with different processor counts, or when both are set with max_thread_count greater than max_merge_count.

index.merge.scheduler.max_merge_count

The maximum number of merges that may be running or waiting on a single shard at once. Defaults to index.merge.scheduler.max_thread_count + 5.

In general, merges are completed concurrently using multiple threads. Generally, each individual merge is single-threaded, but multiple merges can be run in parallel. There are some exceptions and certain merges can utilize multiple threads within the merging operation itself. These threads will be assigned from the same pool as the background merge threads.

The following dynamic settings control multithreaded merging:

index.merge.intra_merge_parallelism_enabled Stack Preview The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Preview This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

Enables or disables intra-merge parallelism for data types that support it (e.g. dense_vector). Defaults to false.