Diversify retriever

The diversify retriever reduces the result set from another retriever by applying a diversification strategy to the top-N results. This is useful when you want to maximize diversity by preventing similar documents from dominating the top results returned from a search. Practical use cases include:

eCommerce applications: Show users a wider variety of products rather than multiple similar items
Retrieval augmented generation (RAG) workflows: Provide more diverse context to the LLM, reducing redundancy in the prompt

The retriever uses MMR (Maximum Marginal Relevance) diversification to discard results that are too similar to each other. Similarity is determined based on the field parameter and any provided query_vector or query_vector_builder.

Note

The ordering of results returned from the inner retriever is preserved.

Parameters

type

(Required, string)

The type of diversification to use. Currently only mmr (maximum marginal relevance) is supported.

field

(Required, string)

The name of the field that will use its values for the diversification process. The field type must be one of:

dense_vector
semantic_text (with dense vector embeddings). For semantic_text fields, you must also provide a query_vector or query_vector_builder. The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Preview since 9.4 This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Preview This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Serverless interfaces and procedures might differ from classic Elastic Stack deployments.

rank_window_size

(Optional, integer)

The maximum number of top-documents the diversify retriever will receive from the inner retriever. Defaults to 10.

retriever

(Required, retriever object)

A single child retriever that provides the initial result set for diversification.

Note

Although some of the inner retriever's results may be removed, the rank and order of the remaining documents is preserved.

query_vector

(Required if the field is a semantic_text type, otherwise optional, array of float, byte or string)

Query vector. Must have the same number of dimensions as the vector field you are searching against. Must be one of:

An array of floats
A hex-encoded byte vector (one byte per dimension; for bit, one byte per 8 dimensions) The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available from 9.0 to 9.3 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.
A base64-encoded vector string. Base64 supports float and bfloat16 (big-endian), byte, and bit encodings depending on the target field type. The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available since 9.4 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. If you provide a query_vector, you cannot also provide a query_vector_builder.

query_vector_builder

(Required if the field is a semantic_text type, otherwise optional, query vector builder object)

Defines a model to build a query vector. If you provide a query_vector_builder, you cannot also provide a query_vector.

lambda

(Required for mmr, float)

A value between 0.0 and 1.0 that controls how similarity is calculated during diversification. Higher values weight the similarity to the query_vector more heavily, lower values weight the diversity more heavily.

size

(Optional, only if mmr is used, integer)

The maximum number of top-N results to return. Defaults to 10.

Example

The following example uses a diversify retriever of type mmr to diversify and return the top three results from the inner standard retriever. The lambda value of 0.7 weights my_dense_field_vector comparisons more heavily than query vector similarity when determining document differences.

GET my_index/_search
{
  "retriever": {
    "diversify": {
      "type": "mmr",
      "field": "my_dense_vector_field",
      "lambda": 0.7,
      "size": 3,
      "query_vector": [0.1, 0.2, 0.3],
      "retriever": {
        "standard": {
          "query": {
            "match": {
              "title": "elasticsearch"
            }
          }
        }
      }
    }
  }
}