How full-text search works

The following diagram illustrates the components of full-text search.

At a high level, full-text search involves the following:

• Text analysis: Analysis consists of a pipeline of sequential transformations. Text is transformed into a format optimized for searching using techniques such as stemming, lowercasing, and stop word elimination. Elasticsearch contains a number of built-in analyzers and tokenizers, including options to analyze specific language text. You can also create custom analyzers.

• Inverted index creation: After analysis is complete, Elasticsearch builds an inverted index from the resulting tokens. An inverted index is a data structure that maps each token to the documents that contain it. It’s made up of two key components:

  • Dictionary: A sorted list of all unique terms in the collection of documents in your index.
  • Posting list: For each term, a list of document IDs where the term appears, along with optional metadata like term frequency and position.

• Relevance scoring: Results are ranked by how relevant they are to the given query. The relevance score of each document is represented by a positive floating-point number called the _score. The higher the _score, the more relevant the document.

  The default similarity algorithm Elasticsearch uses for calculating relevance scores is Okapi BM25, a variation of the TF-IDF algorithm. BM25 calculates relevance scores based on term frequency, document frequency, and document length. Refer to this technical blog post for a deep dive into BM25.

• Full-text search query: Query text is analyzed the same way as the indexed text, and the resulting tokens are used to search the inverted index.

  Query DSL supports a number of full-text queries.

  As of 8.17, ES|QL also supports full-text search functions.