Stack monitoring
Elastic Cloud Hosted lets you manage and configure one or more deployments of the versioned Elastic Stack, hosted on Elastic Cloud. Availability This functionality is generally available and ready for production usage. Elastic Cloud on Kubernetes extends Kubernetes orchestration capabilities to allow you to deploy and manage components of the Elastic Stack. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Elastic Cloud Enterprise is a self-managed orchestration platform for deploying and managing the Elastic Stack at scale. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Self-managed deployments are Elastic Stack deployments managed without the assistance of an orchestrator. Availability This functionality is generally available and ready for production usage. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.
ECH
Generally available
ECK
Generally available
ECE
Generally available
Self-managed
Generally available
Stack monitoring allows you to collect logs and metrics from various Elastic products, including Elasticsearch nodes, Logstash nodes, Kibana instances, APM Server, and Beats in your cluster. You can also collect logs.
All of the monitoring metrics are stored in Elasticsearch, which enables you to easily visualize the data in Kibana. From there you may easily set up Stack monitoring alerts.
Use AutoOps in your Elastic Cloud Hosted deployments, Serverless projects, and ECE, ECK, or self-managed clusters.
AutoOps is a monitoring tool that simplifies cluster management through real-time issue detection and resolution paths, resource utilization visibility, and performance recommendations. Learn more about AutoOps.
To help you make your decision, refer to AutoOps and Stack Monitoring comparison.
Each monitored Elastic Stack component is considered unique in the cluster based on its persistent UUID, which is written to the path.data directory when the node or instance starts.
Monitoring documents are just ordinary JSON documents built by monitoring each Elastic Stack component at a specified collection interval. If you want to alter how these documents are structured or stored, refer to Configuring data streams/indices for monitoring.
You can use Elastic Agent or Metricbeat to collect monitoring data and to ship it directly to the monitoring cluster.
In Elastic Cloud Hosted, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes, Elastic manages the installation and configuration of the monitoring agent for you.
You can collect and ship data directly to your monitoring cluster rather than routing it through your production cluster.
The following diagram illustrates a typical monitoring architecture with separate production and monitoring clusters. This example shows Metricbeat, but you can use Elastic Agent instead.
If you have the appropriate license, you can route data from multiple production clusters to a single monitoring cluster. Learn about the differences between various subscription levels.
In general, the monitoring cluster and the clusters being monitored should be running the same version of the stack. A monitoring cluster cannot monitor production clusters running newer versions of the stack. If necessary, the monitoring cluster can monitor production clusters running the latest release of the previous major version.
Refer to the following topics to learn how to configure stack monitoring:
- Elasticsearch:
- Collecting monitoring data with Elastic Agent (recommended): Uses a single agent to gather logs and metrics. Can be managed from a central location in Fleet.
- Collecting monitoring data with Metricbeat: Uses a lightweight Beats shipper to gather metrics. May be preferred if you have an existing investment in Beats or are not yet ready to use Elastic Agent.
- Collecting log data with Filebeat: Uses a lightweight Beats shipper to gather logs.
- Legacy collection methods for self-managed Elasticsearch: Uses internal exporters to gather metrics. Not recommended. If you have previously configured legacy collection methods, you should migrate to using Elastic Agent or Metricbeat.
- Kibana:
- Collect monitoring data with Elastic Agent (recommended): Uses a single agent to gather logs and metrics. Can be managed from a central location in Fleet.
- Collect monitoring data with Metricbeat: Uses a lightweight Beats shipper to gather metrics. May be preferred if you have an existing investment in Beats or are not yet ready to use Elastic Agent.
- Legacy collection methods for self-managed Kibana: Uses internal exporters to gather metrics. Not recommended. If you have previously configured legacy collection methods, you should migrate to using Elastic Agent or Metricbeat.
Most of these methods require that you configure monitoring of Elasticsearch before monitoring other components.
Logstash:
- Monitoring Logstash with Elastic Agent (recommended): Uses a single agent to gather logs and metrics. Can be managed from a central location in Fleet.
- Monitoring Logstash with legacy collection methods: Use Metricbeat or legacy methods to collect monitoring data from Logstash.
Beats:
Elastic Agents:
- Access monitoring data in Kibana: After you collect monitoring data for one or more products in the Elastic Stack, configure Kibana to retrieve that information and display it in on the Stack Monitoring page.
- Visualizing monitoring data: View health and performance data for Elastic Stack components in real time, as well as analyze past performance.
- Stack monitoring alerts: Configure alerts that trigger based on stack monitoring metrics.
- Configuring monitoring data streams and indices: Adjust the data streams and indices used by stack monitoring.