Get tokens from text analysis

Array of character scripts for which to disable bigrams.

A list of tokens. The filter generates bigrams for these tokens. Either this or the common_words_path parameter is required.

Path to a file containing a list of tokens. The filter generates bigrams for these tokens. This path must be absolute or relative to the config location. The file must be UTF-8 encoded. Each token in the file must be separated by a line break. Either this or the common_words parameter is required.

If true, matches for common words matching are case-insensitive. Defaults to false.

Array of token filters. If a token matches the predicate script in the script parameter, these filters are applied to the token in the order provided.

Character used to separate tokens from payloads. Defaults to |.

Maximum character length of a gram. For custom token filters, defaults to 2. For the built-in edge_ngram filter, defaults to 1.

Minimum character length of a gram. Defaults to 1.

Indicates whether to truncate tokens from the front or back. Defaults to front.

List of elisions to remove. To be removed, the elision must be at the beginning of a token and be immediately followed by an apostrophe. Both the elision and apostrophe are removed. For custom elision filters, either this parameter or articles_path must be specified.

Path to a file that contains a list of elisions to remove. This path must be absolute or relative to the config location, and the file must be UTF-8 encoded. Each elision in the file must be separated by a line break. To be removed, the elision must be at the beginning of a token and be immediately followed by an apostrophe. Both the elision and apostrophe are removed. For custom elision filters, either this parameter or articles must be specified.

Maximum character length, including whitespace, of the output token. Defaults to 255. Concatenated tokens longer than this will result in no token output.

If true, duplicate tokens are removed from the filter’s output. Defaults to true.

One or more .dic files (e.g, en_US.dic, my_custom.dic) to use for the Hunspell dictionary. By default, the hunspell filter uses all .dic files in the <$ES_PATH_CONF>/hunspell/<locale> directory specified using the lang, language, or locale parameter.

Locale directory used to specify the .aff and .dic files for a Hunspell dictionary.

Maximum subword character length. Longer subword tokens are excluded from the output. Defaults to 15.

Minimum subword character length. Shorter subword tokens are excluded from the output. Defaults to 2.

Minimum word character length. Shorter word tokens are excluded from the output. Defaults to 5.

If true, only include the longest matching subword. Defaults to false.

A list of subwords to look for in the token stream. If found, the subword is included in the token output. Either this parameter or word_list_path must be specified.

Path to a file that contains a list of subwords to find in the token stream. If found, the subword is included in the token output. This path must be absolute or relative to the config location, and the file must be UTF-8 encoded. Each token in the file must be separated by a line break. Either this parameter or word_list must be specified.

Path to an Apache FOP (Formatting Objects Processor) XML hyphenation pattern file. This path must be absolute or relative to the config location. Only FOP v1.2 compatible files are supported.

If true, do not match sub tokens in tokens that are in the word list. Defaults to false.

Indicates whether to keep or remove the specified token types.

List of words to keep. Only tokens that match words in this list are included in the output. Either this parameter or keep_words_path must be specified.

If true, lowercase all keep words. Defaults to false.

If true, matching for the keywords and keywords_path parameters ignores letter case. Defaults to false.

Array of keywords. Tokens that match these keywords are not stemmed. This parameter, keywords_path, or keywords_pattern must be specified. You cannot specify this parameter and keywords_pattern.

Path to a file that contains a list of keywords. Tokens that match these keywords are not stemmed. This path must be absolute or relative to the config location, and the file must be UTF-8 encoded. Each word in the file must be separated by a line break. This parameter, keywords, or keywords_pattern must be specified. You cannot specify this parameter and keywords_pattern.

Maximum character length of a token. Longer tokens are excluded from the output. Defaults to Integer.MAX_VALUE, which is 2^31-1 or 2147483647.

If true, the limit filter exhausts the token stream, even if the max_token_count has already been reached. Defaults to false.

Number of buckets to which hashes are assigned. Defaults to 512.

Number of ways to hash each token in the stream. Defaults to 1.

Number of hashes to keep from each bucket. Defaults to 1. Hashes are retained by ascending size, starting with the bucket’s smallest hash first.

A list of token filters to apply to incoming tokens.

Maximum length of characters in a gram. Defaults to 2.

Minimum length of characters in a gram. Defaults to 1.

A list of regular expressions to match.

If true, all substrings matching the pattern parameter’s regular expression are replaced. If false, the filter replaces only the first matching substring in each token. Defaults to true.

Regular expression, written in Java’s regular expression syntax. The filter replaces token substrings matching this pattern with the substring in the replacement parameter.

String used in shingles as a replacement for empty positions that do not contain a token. This filler token is only used in shingles, not original unigrams. Defaults to an underscore (_).

Maximum number of tokens to concatenate when creating shingles. Defaults to 2.

Minimum number of tokens to concatenate when creating shingles. Defaults to 2.

If true, the output includes the original input tokens. If false, the output only includes shingles; the original input tokens are removed. Defaults to true.

If true, the output includes the original input tokens only if no shingles are produced; if shingles are produced, the output only includes shingles. Defaults to false.

A list of mapping rules to use.

If true, stop word matching is case insensitive. For example, if true, a stop word of the matches and removes The, THE, or the. Defaults to false.

If true, the last token of a stream is removed if it’s a stop word. Defaults to true.

Language value, such as _arabic_ or _thai_. Defaults to _english_.

Expands definitions for equivalent synonym rules. Defaults to true.

Sets the synonym rules format.

If true ignores errors while parsing the synonym rules. It is important to note that only those synonym rules which cannot get parsed are ignored. Defaults to the value of the updateable setting.

Used to define inline synonyms.

Used to provide a synonym file. This path must be absolute or relative to the config location.

Provide one or more synonym sets created through the Synonyms Management APIs. Maximum 100 sets per filter.

Expands definitions for equivalent synonym rules. Defaults to true.

Sets the synonym rules format.

If true ignores errors while parsing the synonym rules. It is important to note that only those synonym rules which cannot get parsed are ignored. Defaults to the value of the updateable setting.

Used to define inline synonyms.

Used to provide a synonym file. This path must be absolute or relative to the config location.

Provide one or more synonym sets created through the Synonyms Management APIs. Maximum 100 sets per filter.

If true, the filter produces catenated tokens for chains of alphanumeric characters separated by non-alphabetic delimiters. Defaults to false.

If true, the filter produces catenated tokens for chains of numeric characters separated by non-alphabetic delimiters. Defaults to false.

If true, the filter produces catenated tokens for chains of alphabetical characters separated by non-alphabetic delimiters. Defaults to false.

If true, the filter includes tokens consisting of only numeric characters in the output. If false, the filter excludes these tokens from the output. Defaults to true.

If true, the filter includes tokens consisting of only alphabetical characters in the output. If false, the filter excludes these tokens from the output. Defaults to true.

If true, the filter includes the original version of any split tokens in the output. This original version includes non-alphanumeric delimiters. Defaults to false.

Array of tokens the filter won’t split.

Path to a file that contains a list of tokens the filter won’t split. This path must be absolute or relative to the config location, and the file must be UTF-8 encoded. Each token in the file must be separated by a line break.

If true, the filter splits tokens at letter case transitions. For example: camelCase -> [ camel, Case ]. Defaults to true.

If true, the filter splits tokens at letter-number transitions. For example: j2se -> [ j, 2, se ]. Defaults to true.

If true, the filter removes the English possessive ('s) from the end of each token. For example: O'Neil's -> [ O, Neil ]. Defaults to true.

Array of custom type mappings for characters. This allows you to map non-alphanumeric characters as numeric or alphanumeric to avoid splitting on those characters.

Path to a file that contains custom type mappings for characters. This allows you to map non-alphanumeric characters as numeric or alphanumeric to avoid splitting on those characters.

If true, the filter adjusts the offsets of split or catenated tokens to better reflect their actual position in the token stream. Defaults to true.

If true, the filter produces catenated tokens for chains of alphanumeric characters separated by non-alphabetic delimiters. Defaults to false.

If true, the filter produces catenated tokens for chains of numeric characters separated by non-alphabetic delimiters. Defaults to false.

If true, the filter produces catenated tokens for chains of alphabetical characters separated by non-alphabetic delimiters. Defaults to false.

If true, the filter includes tokens consisting of only numeric characters in the output. If false, the filter excludes these tokens from the output. Defaults to true.

If true, the filter includes tokens consisting of only alphabetical characters in the output. If false, the filter excludes these tokens from the output. Defaults to true.

If true, the filter includes the original version of any split tokens in the output. This original version includes non-alphanumeric delimiters. Defaults to false.

Array of tokens the filter won’t split.

Path to a file that contains a list of tokens the filter won’t split. This path must be absolute or relative to the config location, and the file must be UTF-8 encoded. Each token in the file must be separated by a line break.

If true, the filter splits tokens at letter case transitions. For example: camelCase -> [ camel, Case ]. Defaults to true.

If true, the filter splits tokens at letter-number transitions. For example: j2se -> [ j, 2, se ]. Defaults to true.

If true, the filter removes the English possessive ('s) from the end of each token. For example: O'Neil's -> [ O, Neil ]. Defaults to true.

Array of custom type mappings for characters. This allows you to map non-alphanumeric characters as numeric or alphanumeric to avoid splitting on those characters.

Supported values include: any, common, cyrillic, english, french, german, hebrew, hungarian, polish, romanian, russian, spanish

Maximum subword character length. Longer subword tokens are excluded from the output. Defaults to 15.

Minimum subword character length. Shorter subword tokens are excluded from the output. Defaults to 2.

Minimum word character length. Shorter word tokens are excluded from the output. Defaults to 5.

If true, only include the longest matching subword. Defaults to false.

A list of subwords to look for in the token stream. If found, the subword is included in the token output. Either this parameter or word_list_path must be specified.