Elasticsearch Java Client 9.0.0

While previous versions of the client used to target Java 8, from version 9.0 forward the client will target Java 17.

Impact
Any project that targets Java 8 and needs to import version 9.0 of the elasticsearch Java client.

Action
Projects that import version 9.0 of the elasticsearch Java client as a dependency will have to be updated to be compatible Java 17.

The new version of the client introduces the Rest5Client as an alternative to the legacy RestClient. The RestClient is part of the external dependency elasticsearch-rest-client, which is now optional since an alternative is available.

Impact
Any project updating the elasticsearch Java client.

Action
Either update the code to use the new Rest5Client or import the elasticsearch-rest-client dependency:

// gradle
implementation("org.elasticsearch.client:elasticsearch-rest-client:9.0.0")

<!--maven-->
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-client</artifactId>
    <version>9.0.0</version>
</dependency>

In previous versions of the client, when the server returned number that was both always present and could be null (for example Aggregation results), the client would default to 0 and deserialize the number into a primitive data type instead of an Object. This design choice was changed in this version, changing the fields type to support null and be coherent with the server response.

For more information, check the relevant issue.

Impact
The following classes are affected:

• elasticsearch._types.aggregations.ArrayPercentilesItem
  • value: modified from double to Double, now optional
• elasticsearch._types.aggregations.ExtendedStatsAggregate
  • stdDeviation: modified from double to Double, now optional
  • stdDeviationPopulation: modified from double to Double, now optional
  • stdDeviationSampling: modified from double to Double, now optional
  • sumOfSquares: modified from double to Double, now optional
  • variance: modified from double to Double, now optional
  • variancePopulation: modified from double to Double, now optional
  • varianceSampling: modified from double to Double, now optional
• elasticsearch._types.aggregations.SingleMetricAggregateBase
  • value: modified from double to Double, now optional
• elasticsearch._types.aggregations.StandardDeviationBounds
  • lower: modified from double to Double, now optional
  • lowerPopulation: modified from double to Double, now optional
  • lowerSampling: modified from double to Double, now optional
  • upper: modified from double to Double, now optional
  • upperPopulation: modified from double to Double, now optional
  • upperSampling: modified from double to Double, now optional
• elasticsearch._types.aggregations.StatsAggregate
  • avg: modified from double to Double, now optional
  • max: modified from double to Double, now optional
  • min: modified from double to Double, now optional
• elasticsearch._types.aggregations.StringStatsAggregate
  • avgLength: modified from double to Double, now optional
  • entropy: modified from double to Double, now optional
  • maxLength: modified from int to Integer, now optional
  • minLength: modified from int to Integer, now optional
• elasticsearch._types.aggregations.TTestAggregate
  • value: modified from double to Double, now optional

Action
Steps for mitigating deprecation impact:

• Make sure to handle the possible null value correctly to avoid NullPointerException
• Remove any workaround that was previously suggested to understand whether the 0 returned was actually 0 or null

The Script class used to only support inline string scripts, now it can also support other formats like mustache.

For more information, check the relevant issue.

Impact
The following classes are affected:

• elasticsearch._types.Script
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch._types.ScriptTransform
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch._types.StoredScript
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch.core.msearch_template.TemplateConfig
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch.core.RenderSearchTemplateRequest
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch.core.search.PhraseSuggestCollateQuery
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch.core.SearchTemplateRequest
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch.ingest.ScriptProcessor
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch.watcher.ScriptCondition
  • source: modified from String to elasticsearch._types.ScriptSource
• elasticsearch.ingest.ProcessorBase
  • if_: modified from String to elasticsearch._types.Script

Action
Since the Script builder now supports two variants (string and object), the builder has to be updated to specify the type.
Example with PutScript:

• Old string script

  esClient.putScript(p -> p
      .id("my-script")
      .script(s -> s
          .lang("painless")
          .source("Math.log(_score * 2) + params['my_modifier']")
      )
  );
  

• New string script

  esClient.putScript(p -> p
      .id("my-script")
      .script(s -> s
          .lang("painless")
          .source(so -> so
              .scriptString("Math.log(_score * 2) + params['my_modifier']")
          )
      )
  );
  

• New object script

  esClient.putScript(p -> p
    .id("my-script")
    .script(s -> s
        .lang("mustache")
        .source(so -> so
            .scriptTemplate(st -> st
                .query(q -> q
                    .match(m -> m
                        .field("message")
                        .query("{{query_string}}")
                    )
                )
            )
        )
     )
  );
  

include_named_queries_score is a query parameter that can be enabled for SearchRequest and changes the type of matched_queries in SearchResponse.hits.hits from List<String> to Map<String, Double>. Previous versions of the client didn't support the options, since the json deserialize couldn't know which response type to expect. To support the feature, new versions of the client will always treat matched_queries as a Map<String, Double>, where the values are null in case the original field was returned as a list.

For more information, check the relevant issue.

Impact
The following classes are affected:

• elasticsearch.core.search.Hit
  • matchedQueries: modified from List<String> to Map<String, Double>

Action
Reading matched_queries will be different since it's now a Map, so instead of iterating on the List values now the Map's KeySet has to be iterated to obtain the same result as before.

Breaking change caused by refactoring the ESQL package.

Impact
The following classes are affected:

• elasticsearch.esql.QueryRequest
  • format: modified from elasticsearch.esql.query.EsqlFormat to elasticsearch.esql.EsqlFormat

Action
Change the import from elasticsearch.esql.query.EsqlFormat to elasticsearch.esql.EsqlFormat

Response types should have had a specific body name and getter matching the body type, but in previous versions of the client it was just called valueBody.

Impact
The following classes are affected:

• elasticsearch.ssl.CertificatesResponse
  • valueBody is now certificates
• elasticsearch.snapshot.RepositoryVerifyIntegrityResponse
  • valueBody is now result
• elasticsearch.snapshot.GetRepositoryResponse
  • valueBody is now repositories
• elasticsearch.inference.TextEmbeddingResponse
  • valueBody is now inferenceResult
• elasticsearch.cluster.StateResponse
  • valueBody is now state
• elasticsearch.cat.TransformsResponse
  • valueBody is now transforms
• elasticsearch.cat.ThreadPoolResponse
  • valueBody is now threadPools
• elasticsearch.cat.TemplatesResponse
  • valueBody is now templates
• elasticsearch.cat.TasksResponse
  • valueBody is now tasks
• elasticsearch.cat.SnapshotsResponse
  • valueBody is now snapshots
• elasticsearch.cat.ShardsResponse
  • valueBody is now shards
• elasticsearch.cat.SegmentsResponse
  • valueBody is now segments
• elasticsearch.cat.RepositoriesResponse
  • valueBody is now repositories
• elasticsearch.cat.RecoveryResponse
  • valueBody is now recoveryRecords
• elasticsearch.cat.PluginsResponse
  • valueBody is now plugins
• elasticsearch.cat.PendingTasksResponse
  • valueBody is now pendingTasks
• elasticsearch.cat.NodesResponse
  • valueBody is now nodes
• elasticsearch.cat.NodeattrsResponse
  • valueBody is now nodeAttributes
• elasticsearch.cat.MlTrainedModelsResponse
  • valueBody is now trainedModels
• elasticsearch.cat.MlJobsResponse
  • valueBody is now jobs
• elasticsearch.cat.MlDataFrameAnalyticsResponse
  • valueBody is now dataFrameAnalytics
• elasticsearch.cat.MlDatafeedsResponse
  • valueBody is now datafeeds
• elasticsearch.cat.MasterResponse
  • valueBody is now masters
• elasticsearch.cat.IndicesResponse
  • valueBody is now indices
• elasticsearch.cat.HealthResponse
  • valueBody is now healthRecords
• elasticsearch.cat.FielddataResponse
  • valueBody is now fielddataRecords
• elasticsearch.cat.CountResponse
  • valueBody is now countRecords
• elasticsearch.cat.ComponentTemplatesResponse
  • valueBody is now componentTemplates
• elasticsearch.cat.AllocationResponse
  • valueBody is now allocations
• elasticsearch.cat.AliasesResponse
  • valueBody is now aliases

Action
Replace the valueBody() getter with the specific getter depending on the class used.

indicesBoost and dynamicTemplates were wrongly mapped as List<Map>, but the server doesn't actually accept more than one value in both cases, so the type has been changed to List<NamedValue>.

Impact
The following classes are affected:

• elasticsearch.core.SearchRequest
  • indicesBoost: modified from List<Map<String,Double>> to List<NamedValue<Double>>
• elasticsearch.async_search.SubmitRequest
  • indicesBoost: modified from List<Map<String,Double>> to List<NamedValue<Double>>
• elasticsearch.indices.PutMappingRequest
  • dynamicTemplates: modified from List<Map<String, DynamicTemplate>> to List<NamedValue<DynamicTemplate>>

Action
Change the builder to use the correct type. Example with SearchRequest:

• Old

  esClient.search(s -> s
      .index("*")
      .indicesBoost(Map.of("index", 1.0))
  ,Void.class);
  

• New

  esClient.search(s -> s
      .index("*")
      .indicesBoost(NamedValue.of("index", 1.0))
  ,Void.class);
  

The Elasticsearch server does not accept these requests anymore, or they were replaced with a more specific request.

Impact
The following classes are affected:

• elasticsearch._types.analysis.LanguageAnalyzer: removed
• elasticsearch.nodes.NodeReloadError: removed
• elasticsearch.search_application.list.SearchApplicationListItem: removed
• elasticsearch.inference.*:
  • InferenceRequest, InferenceResponse, InferenceResult, InferenceResultVariant: removed as part of a complete refactor of the Inference API

Action
Find out which are the new requests accepted by the 9.0.0 version of the server.

The reason for these changes is to better reflect what the Elasticsearch server actually accepts/sends, or to fix bugs.

Impact
The following classes are affected:

• elasticsearch._types.analysis.FingerprintAnalyzer
  • maxOutputSize: modified from int to Integer, now optional
• elasticsearch._types.query_dsl.SpanTermQuery
  • value: modified from String to elasticsearch._types.FieldValue
• elasticsearch.async_search.SubmitRequest
  • maxConcurrentShardRequests: modified from Long to Integer
• elasticsearch.core.msearch.RequestItem
  • body: modified from elasticsearch.core.msearch.MultisearchBody to elasticsearch.core.search.SearchRequestBody
• elasticsearch.core.MsearchRequest
  • maxConcurrentSearches: modified from Long to Integer
  • maxConcurrentShardRequests: modified from Long to Integer
• elasticsearch.core.SearchRequest
  • maxConcurrentShardRequests: modified from Long to Integer
• elasticsearch.fleet.FleetSearchRequest
  • maxConcurrentShardRequests: modified from Long to Integer
• elasticsearch.graph.VertexInclude
  • boost: modified from double to Double, now optional
• elasticsearch.security.RoleMappingRule
  • field: modified from elasticsearch.security.FieldRule to util.NamedValue
• elasticsearch._types.mapping.DenseVectorProperty
  • elementType: modified from String to elasticsearch._types.mapping.DenseVectorElementType
  • similarity: modified from String to elasticsearch._types.mapping.DenseVectorSimilarity
• elasticsearch._types.mapping.DynamicTemplate
  • runtime: modified from elasticsearch._types.mapping.Property to elasticsearch._types.mapping.RuntimeField
• elasticsearch._types.mapping.ObjectProperty
  • subobjects: modified from Boolean to elasticsearch._types.mapping.Subobjects
• elasticsearch._types.mapping.TypeMapping
  • subobjects: modified from Boolean to elasticsearch._types.mapping.Subobjects
• elasticsearch.ccr.follow_info.FollowerIndexParameters
  • maxOutstandingReadRequests: modified from int to Long, now optional
  • maxOutstandingWriteRequests: modified from int to Integer, now optional
  • maxReadRequestOperationCount: modified from int to Integer, now optional
  • maxWriteBufferCount: modified from int to Integer, now optional
  • maxWriteRequestOperationCount: modified from int to Integer, now optional
• elasticsearch.ccr.FollowRequest
  • maxOutstandingWriteRequests: modified from Long to Integer
  • maxReadRequestOperationCount: modified from Long to Integer
  • maxWriteBufferCount: modified from Long to Integer
  • maxWriteRequestOperationCount: modified from Long to Integer
• elasticsearch.core.ScriptsPainlessExecuteRequest
  • context: modified from String to elasticsearch.core.scripts_painless_execute.PainlessContext
• elasticsearch.indices.get_data_lifecycle.DataStreamWithLifecycle
  • lifecycle: modified from elasticsearch.indices.DataStreamLifecycle to elasticsearch.indices.DataStreamLifecycleWithRollover
• elasticsearch.ml.TrainedModelDeploymentNodesStats
  • routingState: modified from elasticsearch.ml.TrainedModelAssignmentRoutingTable to elasticsearch.ml.TrainedModelAssignmentRoutingStateAndReason
• elasticsearch.search_application.PutRequest
  • searchApplication: modified from elasticsearch.search_application.SearchApplication to elasticsearch.search_application.SearchApplicationParameters

Action
For types that went from the primitive type to the matching Object, make sure to handle the possible null value.
For msearch.RequestItem, MultisearchBody was just an alias to map SearchRequestBody, so it can be replaced without other changes. For DenseVectorProperty, choose the Enum value that matches the old String.

ElasticsearchClient now can be created with just a few lines of code:

ElasticsearchClient esClient = ElasticsearchClient.of(b -> b
        .host(serverUrl)
        .apiKey(apiKey)
);

The classic version of the builder is still available, both with the new Rest5Client and the legacy RestClient; while the Rest5Client is included with the Java client's dependency, the legacy RestClient must be imported separately:

implementation("org.elasticsearch.client:elasticsearch-rest-client:9.0.0")

A new version of the transport layer, based on the Apache HttpClient 5 is now available to use, as an alternative to the legacy RestClient. More information on the dedicated section: Rest5Client.

Retry logic can now be enabled allowing the BulkIngester to retry operations that failed with error 429 (too many requests), hoping that the error will recover and the request will go through. Users can configure the desired backoff policy using the backoffPolicy() method in the BulkIngester builder:

BulkIngester ingester = BulkIngester.of(b -> b
    .client(client)
    ...
    .listener(listener)
    .flushInterval(1000, TimeUnit.MILLISECONDS)
    .backoffPolicy(BackoffPolicy.constantBackoff(50L, 8))

This is an example of constant backoff, meaning the single failed operation will be retried 8 times every 50 milliseconds.

Some requests in the client require a second parameter to define the result class, for example search, meaning the compiler will complain while the query is being written, which can be annoying. We added overload methods that use Void.class as default type, so that the correct type can be eventually added later into writing the query.
Example with SearchRequest:

• Old:

  esClient.search(s -> s
      .index("my-index")
      .query(q -> q
          .matchAll(m -> m)
      )
  ,Object.class);
  

• New:

  esClient.search(s -> s
      .index("my-index")
      .query(q -> q
          .matchAll(m -> m)
      )
  );
  

Added more setters allowing to build requests with a specific type variant instead of having to use the parent class and then select the desired variant later.

Example with Query, where the query field can now accept a MatchAllQuery (or any other variant) directly:

• Old:

  esClient.search(s -> s
      .index("my-index")
      .query(q -> q
          .matchAll(m -> m)
      )
  );
  

• New:

  esClient.search(s -> s
      .index("my-index")
      .query(MatchAllQuery.of(m -> m))
  );
  

Example with Aggregation, where the aggregations field can now accept AverageAggregation (or any other variant) directly:

• Old:

  // using functional builder shortcut
  esClient.search(s -> s
      .aggregations("agg", a -> a
          .avg(av -> av
          .field("price")
          )
       )
  );
  
  // using Aggregation class builder
  esClient.search(s -> s
      .aggregations("agg", Aggregation.of(ag -> ag
          .avg(av -> av
              .field("price"))
          )
      )
  );
  

• New:

  esClient.search(s -> s
      .aggregations("agg", AverageAggregation.of(av -> av
          .field("price"))
      )
  );