Files are not fully ingested when using autodiscover

By default Filebeat closes files as soon as they are removed. This can cause Filebeat not to ingest the last log lines if files are removed shortly after the last entries were written. This is a common cause of data loss when using Kubernetes autodiscover.

To prevent this from happening, set:

close.on_state_change.removed: false for the Filestream input
close_removed: false for the Log or Container input.

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability Generally available since 9.0 This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes. The hints based autodiscover configuration includes the close.on_state_change.removed setting, set to false by default.

Note

In Filebeat versions 8.x and between versions 9.0.0 - 9.0.7 and 9.1.0 - 9.1.4, this setting isn't specified by default, so you must add it to the configuration manually.

Here is an example of setting close.on_state_change.removed: false when using hints on Kubernetes:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.default_config:
        type: filestream
        id: container-logs-${data.container.id}
        prospector.scanner.symlinks: true
        close.on_state_change.removed: false
        parsers:
          - container: ~
        paths:
          - /var/log/containers/*-${data.container.id}.log