Type to search.

Formatters

Serverless

Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud.

Availability

  • Generally available

    This functionality is generally available and ready for production usage.

Serverless interfaces and procedures might differ from classic Elastic Stack deployments.

Stack

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats.

Availability

  • Generally available

    This functionality is generally available and ready for production usage.

Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments.

This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

Our log formatters allow you to change the way various logging frameworks log to IO (file/console) to use ECS json.

ECS aware message templates

Our log formatters allow you to set ECS fields directly from the message template using properties that adhere to the https://messagetemplates.org/ format.

C# 
Log.Information("The time is {TraceId}", "my-trace-id");

Will directly override trace.id on the resulting ECS json document.

All supported ECS message template properties are available as constants under the LogTemplateProperties static class. For example LogTemplateProperties.TraceId will return "TraceId".

Ingest ECS log files

Serverless Unavailable

Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud.

Availability

  • Unavailable

    This functionality is not available in Elastic Cloud Serverless.

If you are using one of our formatter libraries to log to file or stdout/stderr you can use the following options to get these logs into Elasticsearch or Elastic Cloud:

Note

We also support writing logs directly to Elasticsearch or Elastic Cloud. See our various data shipper integrations

  1. Follow the Filebeat quick start
  2. Add the following configuration to your filebeat.yaml file.

For Filebeat 7.16+

YAML 
filebeat.inputs:
- type: filestream
  paths: /path/to/logs.json
  parsers:
    - ndjson:
      overwrite_keys: true
      add_error_key: true
      expand_keys: true

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  1. Use the filestream input to read lines from active log files.
  2. Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
  3. Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
  4. Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
  5. Processors enhance your data. See processors to learn more.

For Filebeat < 7.16

YAML 
filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
  1. Make sure your application logs to stdout/stderr.
  2. Follow the Run Filebeat on Kubernetes guide.
  3. Enable hints-based autodiscover (uncomment the corresponding section in filebeat-kubernetes.yaml).
  4. Add these annotations to your pods that log using ECS loggers. This will make sure the logs are parsed appropriately.
YAML 
annotations:
  co.elastic.logs/json.overwrite_keys: true
  co.elastic.logs/json.add_error_key: true
  co.elastic.logs/json.expand_keys: true
  1. Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
  2. Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
  3. Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
  1. Make sure your application logs to stdout/stderr.
  2. Follow the Run Filebeat on Docker guide.
  3. Enable hints-based autodiscover.
  4. Add these labels to your containers that log using ECS loggers. This will make sure the logs are parsed appropriately.
YAML 
labels:
  co.elastic.logs/json.overwrite_keys: true
  co.elastic.logs/json.add_error_key: true
  co.elastic.logs/json.expand_keys: true
  1. Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.
  2. Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
  3. Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.

For more information, see the Filebeat reference.