Get data into Streams

This page shows you how to send new data to Streams or use Streams to work with existing data:

Ingest new data: Use wired streams to send logs to a managed endpoint. This option suits new deployments, custom logs, and mixed-format sources.
Work with existing data: Use classic streams to work with data already flowing into Elasticsearch. No migration or configuration changes are required.

Before you begin

Streams requires the following permissions:

Serverless

Streams requires one of the following Elastic Cloud Serverless roles:

Admin: Able to manage all Streams
Editor/Viewer: Has limited access to Streams, cannot perform all actions

Stack

To manage all streams, you need the following permissions:

Cluster permissions: manage_index_templates, manage_ingest_pipelines, manage_pipeline, read_pipeline
Data stream level permissions: read, write, create, manage, monitor, manage_data_stream_lifecycle, read_failure_store, manage_failure_store, manage_ilm.

To view streams, you need the following permissions:

Data stream level: read, view_index_metadata, monitor

For more information, refer to Cluster privileges and Granting privileges for data streams and aliases.

Ingest new data with wired streams

Wired streams send documents to a managed endpoint, from which you can route data into child streams based on partitioning rules. Child streams inherit mappings, lifecycle settings, and processors from the parent.

Select the endpoint that matches your deployment:

Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Preview This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Serverless interfaces and procedures might differ from classic Elastic Stack deployments. The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Preview since 9.4 This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Use logs.otel for OTel-native data or for a consistent, normalized format. Streams translates ECS field names to OTel equivalents automatically. Use logs.ecs when your data already uses ECS field names and you want to preserve them without transformation.
The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Preview from 9.2 to 9.3 This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. Use logs. This is the only wired-stream endpoint available in these versions. Replace logs.otel with logs in the following examples.

Configure your shipper to use the endpoint:

OpenTelemetry

processors:
  transform/logs-streams:
    log_statements:
      - context: resource
        statements:
          - set(attributes["elasticsearch.index"], "logs.otel")
service:
  pipelines:
    logs:
      receivers: [myreceiver]
      processors: [transform/logs-streams]
      exporters: [elasticsearch, otlp]

Set to logs.otel or logs.ecs (Serverless and Stack 9.4+), or logs (Stack 9.2–9.3)
Works with any logs receiver
Works with either

Filebeat

filebeat.inputs:
  - type: filestream
    id: my-filestream-id
    index: logs.otel
    enabled: true
    paths:
      - /var/log/*.log

# No need to install templates for wired streams
setup:
  template:
    enabled: false

output.elasticsearch:
  hosts: ["<elasticsearch-host>"]
  api_key: "<your-api-key>"

Set to logs.otel or logs.ecs (Serverless and Stack 9.4+), or logs (Stack 9.2–9.3)

Logstash

output {
  elasticsearch {
    hosts => ["<elasticsearch-host>"]
    api_key => "<your-api-key>"
    index => "logs.otel"
    action => "create"
  }
}

Set to logs.otel or logs.ecs (Serverless and Stack 9.4+), or logs (Stack 9.2–9.3)

Fleet

Use the Custom Logs (Filestream) integration to send data to wired streams:

Find Fleet in the navigation menu or use the global search field.
Select the Settings tab.
Under Outputs, find the output you want to use and select the icon.
Turn on Write to logs streams.
Add the Custom Logs (Filestream) integration to an agent policy.
Enable the Use the "logs" data stream setting under Change defaults.
Under Where to add this integration, select an agent policy that uses the output configured in step 4.

API

Send data to the endpoint using the Bulk API:

POST /logs.otel/_bulk
{ "create": {} }
{ "@timestamp": "2025-05-05T12:12:12", "body": { "text": "Hello world!" }, "resource": { "attributes": { "host.name": "my-host-name" } } }
{ "create": {} }
{ "@timestamp": "2025-05-05T12:12:12", "message": "Hello world!", "host.name": "my-host-name" }

Verify data is flowing

After configuring your data source, confirm data is appearing in Discover.

For wired streams, you first need to make the index pattern available:

Manually create a data view for the wired streams index pattern (logs,logs.*).
Add the wired streams index pattern (logs,logs.*) to the observability:logSources Kibana advanced setting, which you can open from the navigation menu or by using the global search field.

Once data appears in Discover, you're ready to start organizing, parsing, and configuring retention for your streams.

Query unmapped fields

Wired streams can contain fields stored in _source that are not explicitly mapped. By default, ES|QL returns an error when a query references an unmapped field. To make unmapped fields queryable, add SET unmapped_fields = "LOAD"; at the start of your ES|QL query:

SET unmapped_fields = "LOAD";
FROM logs.otel
| WHERE my_custom_field == "value"

When LOAD is set, unmapped fields are loaded from _source as keyword fields, or treated as null if absent from _source.

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Preview since 9.5 This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. When you query a wired stream and the ES|QL editor detects an unknown column error, a Load unmapped fields quick fix action is available. Select it to apply this setting automatically.

For a conceptual overview and use cases, refer to Unmapped fields. For Kibana editor behavior, refer to Handle unmapped fields with SET unmapped_fields.

Work with existing data with classic streams

Classic streams let you use the Streams UI to extract fields and configure data retention for data that's already being ingested into Elasticsearch without additional configuration.

Warning

Do not reroute or migrate existing data streams to the logs, logs.otel, or logs.ecs endpoints. Classic streams work with your existing data streams in place. No migration is required.

Classic streams:

Are based on existing data streams, index templates, and component templates.
Can follow the data retention policy set in the existing index template.
Do not support hierarchical inheritance or cascading configuration updates.

Open classic streams from the following places in Kibana:

Select Streams from the navigation menu or use the global search field, and find the data stream in the Streams table.
Open the data stream for a specific document from Discover. To do this:
1. From the Documents table, select the icon for a document to expand the details flyout.
2. Under Stream, select the link to the stream (for example, logs-generic.default). The Streams UI opens filtered to the stream that contains the document.

Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability Preview This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Serverless interfaces and procedures might differ from classic Elastic Stack deployments. The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Preview since 9.1 This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. You can also access Streams features using the Streams API. Refer to the Streams API documentation for more information.

Next steps

Once your data is flowing into Streams, you can start organizing and enriching it:

Organize your data: Use partitioning to route data subsets into dedicated child streams with independent retention and processing rules. Partitioning is only available for wired streams.
Parse and process: Build a processing pipeline to extract structured fields from raw log messages using AI-generated or manually configured processors.