Type to search.

Troubleshoot an out-of-memory Elastic Agent

Serverless Observability

Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud.

Availability

  • Generally available

    This functionality is generally available and ready for production usage.

Stack

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats.

Availability

  • Generally available

    This functionality is generally available and ready for production usage.

Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments.

This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

Elastic Agent

Elastic Agent can run in OpenTelemetry mode to retrieve traces, metrics, and logs from your infrastructure and applications, and forward them to Elastic Observability.

Availability

  • Generally available

    This functionality is generally available and ready for production usage.

This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

If your Elastic Agent pods terminate with an OOMKilled status, this usually indicates sustained memory pressure or potentially a memory leak due to an introduced regression or a bug. You can use the Performance Profiler (pprof) extension to collect and analyze memory profiles, helping you identify the root cause of the issue.

If you're running the Collector in Kubernetes and experiencing resource allocation issues, refer to Insufficient resources in Kubernetes for troubleshooting steps.

These symptoms typically indicate that the Elastic Agent is experiencing a memory-related failure:

  • Elastic Agent pod restarts with an OOMKilled status in Kubernetes.
  • Memory usage steadily increases before the crash.
  • The Collector's logs don't show clear errors before termination.

For more detailed diagnostics, refer to Enable debug logging.

Turn on runtime profiling using the pprof extension and then gather memory heap profiles from the affected pod:

  1. Enable pprof in the Collector

    Edit the Elastic Agent Daemonset configuration and include the pprof extension:

    YAML 
    exporters:
  ...
processors:
  ...
receivers:
  ...
extensions:
  pprof:

service:
  extensions:
   - pprof
   - ...
  pipelines:
    metrics:
      receivers: [ ... ]
      processors: [ ... ]
      exporters: [ ... ]

    Restart the Collector after applying these changes. When the Daemonset is deployed again, spot the pod that is getting restarted.

  2. Access the affected pod and collect a heap dump

    When a pod starts exhibiting high memory usage or restarts due to OOM, run the following to enter a debug shell:

    JSON 
    
kubectl debug -it <collector-pod-name> --image=ubuntu:latest

    In the debug container:

    JSON 
    
apt update
apt install -y curl
curl http://localhost:1777/debug/pprof/heap > heap.out

  3. Copy the heap file from the pod

    From your local machine, copy the heap file using:

    Shell 
    kubectl cp <collector-pod-name>:heap.out ./heap.out -c <debug-container-name>
    Note

    Replace <debug-container-name> with the name assigned to the debug container. Without the -c flag, Kubernetes will show the list of available containers.

  4. Convert the heap profile for analysis

    You can now generate a visual representation, for example PNG:

    Shell 
    go tool pprof -png heap.out > heap.png

To improve the effectiveness of memory diagnostics and reduce investigation time, consider the following:

  • Collect multiple heap profiles over time (for example, every few minutes) to observe memory trends before the crash.

  • Automate heap profile collection at intervals to observe trends over time.