Type to search.

LLM performance matrix for the Observability AI Assistant

Serverless

Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud.

Availability

  • Generally available

    This functionality is generally available and ready for production usage.

Serverless interfaces and procedures might differ from classic Elastic Stack deployments.

Stack 9.2+

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats.

Availability

  • Generally available since 9.2

    This functionality is generally available and ready for production usage.

Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments.

This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

This page summarizes internal test results comparing large language models (LLMs) across Observability AI chat use cases. These ratings only apply if you're using AI Assistant. For Agent Builder, refer to recommended models.

Warning Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Generally available This functionality is generally available and ready for production usage. Serverless interfaces and procedures might differ from classic Elastic Stack deployments. The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available since 9.4 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

The Elastic AI Assistant for Observability and Search is deprecated. The Elastic AI Agent is now the default chat experience in Observability. To switch back to the AI Assistant, go to GenAI settings.

Rating legend

For each category, the evaluation framework produces a score between 0–100 based on the criteria defined. This score is then converted to a rating using the following legend:

Score Rating
84%–100% Excellent: Highly accurate and reliable for the use case.
75%–83% Great: Strong performance with minor limitations.
45%–74% Good: Possibly adequate for many use cases but with noticeable tradeoffs.
0%–44% Poor: Significant issues; not recommended for production for the use case.

Recommended models are those rated Excellent or Great for the particular use case.

Proprietary models

Models from third-party LLM providers.

Provider Model Alert questions APM questions Contextual insights Documentation retrieval Elasticsearch operations ES|QL generation Execute connector Knowledge retrieval
Amazon Bedrock Claude Sonnet 3.5 Excellent Excellent Excellent Excellent Excellent Excellent Good Excellent
Amazon Bedrock Claude Sonnet 3.7 Excellent Excellent Excellent Excellent Excellent Excellent Great Excellent
Amazon Bedrock Claude Sonnet 4 Excellent Excellent Excellent Excellent Excellent Excellent Great Excellent
Amazon Bedrock Claude Sonnet 4.5 Excellent Excellent Excellent Excellent Excellent Excellent Good Excellent
Google Gemini Gemini 2.0 Flash Excellent Good Excellent Excellent Excellent Good Good Excellent
Google Gemini Gemini 2.5 Flash Excellent Good Excellent Excellent Excellent Great Good Excellent
Google Gemini Gemini 2.5 Pro Excellent Great Excellent Excellent Excellent Great Good Excellent
OpenAI GPT-4.1 Excellent Excellent Excellent Excellent Excellent Great Good Excellent
OpenAI GPT-4.1 Mini Excellent Great Excellent Excellent Excellent Great Good Excellent
OpenAI GPT-5 Excellent Excellent Excellent Excellent Excellent Excellent Good Excellent
OpenAI GPT-5.2 Excellent Great Excellent Excellent Excellent Great Good Excellent

Open-source models

Serverless Preview

Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud.

Availability

  • Preview

    This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply.

Serverless interfaces and procedures might differ from classic Elastic Stack deployments.

Stack Preview 9.2+

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats.

Availability

  • Preview since 9.2

    This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply.

Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments.

This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

Models you can deploy and manage yourself.

Provider Model Alert questions APM questions Contextual insights Documentation retrieval Elasticsearch operations ES|QL generation Execute connector Knowledge retrieval
DeepSeek DeepSeek-V3.1 Excellent Excellent Excellent Excellent Excellent Great Great Excellent
Google DeepMind Gemma-3-27b-it Excellent Good Great Great Excellent Good Great Excellent
OpenAI gpt-oss-20b Poor Poor Great Poor Good Poor Good Good
OpenAI gpt-oss-120b Excellent Poor Great Great Excellent Good Good Excellent
Meta Llama-3.3-70B-Instruct Excellent Good Great Excellent Excellent Good Good Excellent
Meta Llama-4-Maverick-17B-128E-Instruct Great Good Great Excellent Excellent Good Good Great
Mistral Mistral-Small-3.2-24B-Instruct-2506 Excellent Poor Great Great Excellent Good Good Excellent
Alibaba Cloud Qwen2.5-72b-Instruct Excellent Good Great Excellent Excellent Good Good Excellent
Note

Llama-3.3-70B-Instruct and Qwen2.5-72b-Instruct were tested with simulated function calling.

You can run the Observability AI evaluation framework against any model, and use it to benchmark a custom or self-hosted model against the use cases in the matrix. Refer to the evaluation framework README for setup and usage details.

For consistency, all ratings in this matrix were generated using Gemini 2.5 Pro as the judge model (specified through the --evaluateWith flag). Use the same judge when evaluating your own model to ensure comparable results.