Elastic Agent to Logstash to Kafka to Logstash to Elasticsearch: Kafka as middleware message queue
Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability This functionality is generally available and ready for production usage. Serverless interfaces and procedures might differ from classic Elastic Stack deployments. The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.
Serverless
Generally available
Stack
Generally available
- Ingest model
-
Control path: Elastic Agent to Fleet to Elasticsearch
Data path: Elastic Agent to Logstash to Kafka to Logstash to Elasticsearch: Kafka as middleware message queue.Logstash reads data from Kafka and routes it to Elasticsearch clusters, other destinations, or both
- Use when
- You are standardizing on Kafka as middleware message queue between Elastic Agent and Elasticsearch
- Notes
- The transformation from raw data to Elastic Common Schema (ECS) and any other enrichment can be handled by Logstash as described in Elastic Agent to Logstash (for enrichment) to Elasticsearch.
Info on Elastic Agent and agent integrations:
Info on Logstash and Logstash Kafka plugins:
- Logstash Reference
- Logstash Elastic Agent input
- Logstash Kafka input
- Logstash Kafka output
- Logstash Elasticsearch output
Info on Elasticsearch: