Observability known issues

Applies to: Elastic Stack 9.5.0

Details

Do not upgrade private location Elastic Agents that run Synthetics browser (journey) monitors to 9.5.0. Keep those agents on 9.4.x (for example 9.4.4) and wait for 9.5.1. It is supported to run an older Elastic Agent version (such as 9.4.4) against a 9.5.0 Elastic Stack. Lightweight monitors (HTTP, TCP, and ICMP) on the same private location and monitors on Elastic-managed locations are not affected.

On 9.5.0, browser monitors on Fleet-managed private locations fail for two independent Elastic Agent defects:

1. Missing Playwright browser binaries in the elastic-agent-complete image. Journeys fail at launch with browserType.launch: Executable doesn't exist at .../chromium_headless_shell-<rev>/.... For more information, check elastic-agent#15993.

2. OTel Heartbeat runtime rejects browser monitor config and reports the component as permanently failed with missing required field accessing 'heartbeat.monitors.0.schedule'. For more information, check elastic-agent#15968.

Both issues are addressed in 9.5.1. There is no supported workaround on 9.5.0 that restores browser monitors without changing the agent version.

Applies to: Elastic Stack 9.3.0, 9.3.1, 9.3.2, 9.3.3, 9.3.4

Details

Upgrading from 9.2.x to 9.3.x can fail if a rule (including Observability alerting rules) has a connector action whose parameter values are larger than 32,766 bytes. Common examples include email message bodies or HTML templates, large webhook payloads, or Slack messages built from verbose templates.

During the upgrade, Kibana migrates rule saved objects to a new internal mapping. Any oversized action parameter value causes the migration to abort with an error similar to:

Flattened field [alert.actions.params] contains one immense field whose keyed encoding is longer than the allowed max length of 32766 bytes

Workaround

Upgrade to 9.3.5 or 9.4.2.

If the upgrade has failed with this error, identify rules that use connectors with large content (particularly email, webhook, and Slack connectors) and shorten the action parameter values, such as message bodies or HTML templates. Then retry the upgrade.

For more information, refer to #268982.

Resolved

This issue is resolved in Elastic Stack 9.3.5 and 9.4.2.

Applies to: Elastic Stack 9.2.7, 9.2.8, 9.3.2, 9.3.3

Details

Some Observability alerts send recovery notifications (for example, Slack, email, or webhook) but remain active in Kibana instead of transitioning to recovered.

Action

Manually untrack stale alerts or upgrade to Elastic Stack 9.3.4 or 9.4.0.

Resolved

This issue is resolved in Elastic Stack 9.3.4 and 9.4.0.

Applies to: All Elastic Stack versions

Details

Browser monitors created through the Kibana Synthetics UI or the public APIs that are not using projects fail to run on private locations when inline scripts contain JavaScript template literals (`${variable}`).

A fix for this issue is expected in Elastic Stack 9.4.

For more information, check Issue #248.

Workaround

Use regular JavaScript string concatenation instead of template literals.

For example, instead of:

step('Go to page', async () => {
  await page.goto(`${params.basePath}${params.route}`);
});

Use:

step('Go to page', async () => {
  await page.goto(params.basePath + params.route);
});

Alternatively, you can switch to project monitors.

Applies to: Elastic Stack 8.19.5 and later

Details

When upgrading from a version before 8.19.5 to 8.19.5 or later, Kibana might unintentionally delete package policies for Synthetics project monitors during package policy cleanup. In deployments with many project monitors (10,000+ package policies in a single space), attempts to recreate deleted policies can fail and leave your instance in a broken state.

A fix for this issue was implemented in #248762.

Detecting missing package policies

The following queries require superuser privileges. If you don't have superuser privileges and have pending monitors after upgrading, try making a dummy update to one of them through the UI to resolve the issue.

1. Check the task state:

   GET .kibana_*/_search
   {
     "size": 1,
     "query": {
       "bool": {
         "filter": [
           {
             "match_phrase": {
               "task.taskType": "Synthetics:Sync-Private-Location-Monitors"
             }
           }
         ]
       }
     }
   }
   

   If hasAlreadyDoneCleanup is false in the response and the task is timing out, you need to manually update the task state as shown in the following example. If hasAlreadyDoneCleanup is true, this issue most likely isn't causing your problem.

   POST .kibana_task_manager_9.3.0_001/_update/task:Synthetics:Sync-Private-Location-Monitors-single-instance
   {
     "doc": {
       "task": {
         "state": "{\"hasAlreadyDoneCleanup\":true, <copy the rest of the state from above response> }"
       }
     }
   }
   

2. Get the total count of monitors by summing the allLocs and allLocsMulti aggregation values:

   GET .kibana*/_search
   {
     "size": 0,
     "query": {
       "bool": {
         "filter": [
           {
             "terms": {
               "type": [
                 "synthetics-monitor",
                 "synthetics-monitor-multi-space"
               ]
             }
           }
         ],
         "must_not": [
           {
             "terms": {
               "synthetics-monitor.locations.id": [
                 "asia-northeast1-a",
                 "asia-south1-a",
                 "asia-southeast1-a",
                 "australia-southeast1-a",
                 "europe-west2-a",
                 "europe-west3-a",
                 "northamerica-northeast1-a",
                 "southamerica-east1-a",
                 "us-east4-a",
                 "us-west1-a"
               ]
             }
           },
           {
             "terms": {
               "synthetics-monitor-multi-space.locations.id": [
                 "asia-northeast1-a",
                 "asia-south1-a",
                 "asia-southeast1-a",
                 "australia-southeast1-a",
                 "europe-west2-a",
                 "europe-west3-a",
                 "northamerica-northeast1-a",
                 "southamerica-east1-a",
                 "us-east4-a",
                 "us-west1-a"
               ]
             }
           }
         ]
       }
     },
     "aggs": {
       "allLocs": {
         "value_count": {
           "field": "synthetics-monitor-multi-space.locations.id"
         }
       },
       "allLocsMulti": {
         "value_count": {
           "field": "synthetics-monitor.locations.id"
         }
       }
     }
   }
   

3. Get the count of package policies:

   GET .kibana*/_search?track_total_hits=true
   {
     "size": 0,
     "_source": {
       "excludes": "ingest-package-policies.inputs"
     },
     "query": {
       "bool": {
         "filter": [
           {
             "term": {
               "type": "ingest-package-policies"
             }
           },
           {
             "term": {
               "ingest-package-policies.package.name": "synthetics"
             }
           }
         ]
       }
     }
   }
   

4. Compare the counts. If the package policy count doesn't match the monitor count, you have missing package policies and need to apply the workaround.

Workaround

Trigger package policy recreation using one of the following methods:

• Make a dummy update to the affected project monitors through the UI (for example, add a tag).
• Make a dummy edit to the private location configuration. Like editing the name of private location, this regenerates all package policies for that location's monitors.
• Push a dummy project monitor update, like adding a tag to the project monitor config and run a npm run push.

Applies to: Elastic Stack

Details

Users with Helm version 3.18.5 or later will see an error similar to the following when using the Kubernetes OpenTelemetry quickstart onboarding flow:

Error: values don't meet the specifications of the schema(s) in the following chart(s):
opentelemetry-operator:
"file:///values.schema.json#" is not valid against metaschema: jsonschema validation failed with 'https://json-schema.org/draft/2019-09/schema#'
- at '': 'allOf' failed
  - at '/properties/manager': 'allOf' failed
    - at '/properties/manager/properties/featureGates': 'allOf' failed
      - at '/properties/manager/properties/featureGates/examples': got string, want array

For more information, check:

• Issue 232667
• Issue 9535

Workaround

Downgrade Helm to version 3.18.4.

Applies to: Elastic Stack 9.x

The Elastic Managed LLM may be automatically selected as your default connector because of existing connector selection logic. This can occur if you had not previously specified a connector for any of the following reasons:

• You only had one connector available and it was always automatically picked for your conversations.
• You had multiple connectors available but didn’t make a specific selection and used the automatically picked connector for your conversations.
• You previously selected a connector but cleared your browser's local storage or switched browsers or devices.

And:

• All of your existing connector names come after the “Elastic Managed LLM connector" when sorted alphabetically.

For more information, check #2088

Applies to: Elastic Stack 9.0.0, 9.0.1, 9.0.2

Details

The Observability AI assistant gets stuck in a loop when calling the execute_connector function. Instead of completing queries, it times out with the error message Failed to parse function call arguments when converting messages for inference: SyntaxError: Unexpected non-whitespace character after JSON at position 72 and Error: Tool call arguments for execute_connector (...) were invalid.

Applies to: Elastic Stack 9.0.0

Details

The semantic text migration can cause excessive traffic to a cluster and might eventually cause the Kibana instance to crash due to OOM, together with increase of requests to Elasticsearch & ML nodes.

The problem can occur when there is one or more empty text Knowledge Base documents.

The migration script does not handle this scenario and will indefinitely update the same document.

Because the document update involves semantic_text an ML node is kept warm further increasing the costs.

The issue involves semantic_text field type (and the semantic_text migration which is causing this issue), introduced in the knowledge base feature in 8.17.

Workaround

1. Pause the Kibana instance if possible. If not possible, skip this step.

2. Run a dry run query to identify if you have empty Knowledge Base documents. If you have at least 1 hit, you can be affected by the problem.

   GET .kibana-observability-ai-assistant-kb/_search
   {
     "query": {
       "bool": {
         "must": [{ "exists": { "field": "text" }}],
         "must_not": [ { "wildcard": { "text": "*" } }
         ]
       }
     }
   }
   

3. Execute the deletion. For extra safety, you might want to trigger a snapshot before executing it.

   POST .kibana-observability-ai-assistant-kb/_delete_by_query
   {
     "query": {
       "bool": {
         "must": [{ "exists": { "field": "text" }}],
         "must_not": [ { "wildcard": { "text": "*" } }
         ]
       }
     }
   }
   

For more information, check:

• #220339
• #220342

Applies to: Elastic Stack 9.x, 8.x

Details

After upgrading a cluster, Collector and Symbolizer endpoints may not be configured even when Universal Profiling is enabled in Kibana and the "Add data" instructions appear on the Universal Profiling landing page.

Workaround

1. Run the following query to retrieve the ids of the elastic-universal-profiling-collector and elastic-universal-profiling-symbolizer package policies:

   GET kbn:/api/fleet/package_policies
   

2. Delete the package policies:

   DELETE kbn:/api/fleet/package_policies/<elastic-universal-profiling-collector-id>?force=true
   DELETE kbn:/api/fleet/package_policies/<elastic-universal-profiling-symbolizer-id>?force=true
   

Applies to: Elastic Stack 9.1.6, 9.2.0

Details

Users trying to collect logs and host metrics using the Elastic distribution of the OTel collector will see error when using the OpenTelemetry quickstart onboarding flow:

> sudo ./otelcol --config otel.yml

Starting in otel mode
failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):

'exporters' error reading configuration for "otlp/ingest": decoding failed due to the following error(s):

'sending_queue' decoding failed due to the following error(s):

'batch' decoding failed due to the following error(s):

'' has invalid keys: flush_interval

Workaround

To work around this issue, manually update the configuration of the generated otel.yaml file to replace incorrect key flush_interval with the correct key flush_timeout.

batch:
  flush_timeout: 1s

Applies to: Elastic Stack 9.1, 9.2, 9.3

Details

Due to a bug in how Elasticsearch processes empty keyword fields encoded with SMILE, Symbolizer fails to store symbols for private executables. Elasticsearch versions affected: 9.1.4+ Fixed in the following Elasticsearch releases:

• 9.1.11
• 9.2.8
• 9.3.3