--- title: Custom AWS Log Integration description: The custom AWS input integration offers users two ways to collect logs from AWS: from an S3 bucket (with or without SQS notification) and from CloudWatch... url: https://www.elastic.co/docs/reference/integrations/aws_logs products: - Elastic integrations applies_to: - Serverless Observability projects: Generally available - Serverless Security projects: Generally available - Elastic Stack: Generally available since 9.0 --- # Custom AWS Log Integration | | | |-------------------------------------------------------------------------------------------------|--------------------------------| | **Version** | 1.8.3 ([View all](#changelog)) | | **Subscription level**[What's this?](https://www.elastic.co/subscriptions) | Basic | | **Developed by**[What's this?](https://www.elastic.co/docs/reference/integrations/developed-by) | Elastic | | **Ingestion method(s)** | AWS CloudWatch, AWS S3 | | **Minimum Kibana version(s)** | 9.0.08.16.5 | The custom AWS input integration offers users two ways to collect logs from AWS: from an [S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html) (with or without SQS notification) and from [CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html). Custom ingest pipelines may be added by adding the name to the pipeline configuration option, creating custom ingest pipelines can be done either through the API or the *Ingest Node Pipeline UI* (use the [global search field](https://docs-demo.elastic.dev/docs/explore-analyze/find-and-organize/find-apps-and-objects) to search for "ingest pipelines"). ## Collecting logs from S3 bucket When collecting logs from S3 bucket is enabled, users can retrieve logs from S3 objects that are pointed to by S3 notification events read from an SQS queue or directly polling list of S3 objects in an S3 bucket. The use of SQS notification is preferred: polling list of S3 objects is expensive in terms of performance and costs and should be preferably used only when no SQS notification can be attached to the S3 buckets. This input integration also supports S3 notification from SNS to SQS. You can enable SQS notification method by setting `queue_url` and `number_of_workers` configuration values. You can enable S3 bucket list polling method by setting `bucket_arn`, `access_point_arn` or `non_aws_bucket_name` configuration values and `number_of_workers` value. `queue_url`, `bucket_arn`, `access_point_arn` and `non_aws_bucket_name` cannot be set at the same time and at least one of these value must be set. To access SQS and S3, these [specific AWS permissions](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#_aws_permissions_2) are required. ## Collecting logs from CloudWatch When collecting logs from CloudWatch is enabled, users can retrieve logs from all log streams in a specific log group. `filterLogEvents` AWS API is used to list log events from the specified log group. Amazon CloudWatch Logs can be used to store log files from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sources. To access aws-cloudwatch, these [specific AWS permissions](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-cloudwatch.html#_aws_permissions) are required. ## Changelog | Version | Details | Minimum Kibana version | |-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------| | **1.8.3** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/15507))Remove fixed value from event.dataset mapping. | 9.0.08.16.5 | | **1.8.2** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/14733))Add description on lastSync start_position configuration for CloudWatch. | 9.0.08.16.5 | | **1.8.1** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/14416))Fix handling of SQS worker count configuration. | 9.0.08.16.5 | | **1.8.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/12645))Add support to configure start_timestamp and ignore_older configurations for AWS S3 backed inputs. | 9.0.08.16.5 | | **1.7.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/12637))Add support for Kibana `9.0.0` | 9.0.08.16.2 | | **1.6.1** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/12223))Add missing category. | 8.16.2 | | **1.6.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/12154))Add support for Access Point ARN when collecting logs via the AWS S3 Bucket. | 8.16.2 | | **1.5.2** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/12100))Fix dupplicated keys buffer_size, fips_enabled, in agent template. | 8.16.0 | | **1.5.1** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/11891))Add required permissions for Custom AWS Logs. | 8.16.0 | | **1.5.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/11484))Add option to check linked accounts when using log group prefixes to derive matching log groups | 8.16.0 | | **1.4.2** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/11817))Add ingest pipeline input option back, which was removed in 1.4.1. | 8.12.0 | | **1.4.1** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/11681))Remove ingest pipeline input option. | 8.12.0 | | **1.4.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/10789))Update file_selectors field to be able to receive multiline configuration | 8.12.0 | | **1.3.1** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/10480))Update max_number_of_messages parameter description | 8.12.0 | | **1.3.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/9730))Improve clarity and validity of system testing. | 8.12.0 | | **1.2.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/9512))Add a custom settings variable to allow passing additional settings to the input. | 8.12.0 | | **1.1.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/9019))Update the package format_version to 3.0.2 and enable 'secret' for the sensitive fields | 8.12.0 | | **1.0.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/8991))Release package as GA. | 8.0.0 | | **0.6.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/8041))Update the package format_version to 3.0.0. | 8.0.0 | | **0.5.1** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/7481))Remove duplicated number_of_workers settings | 8.0.0 | | **0.5.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/6841))Add permissions to reroute events to logs-*-* for generic datastream | 8.0.0 | | **0.4.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/6081))Add multiline support for using s3 input | 8.0.0 | | **0.3.3** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/5123))Added categories and/or subcategories. | 8.0.0 | | **0.3.2** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/4917))Add required field number of workers to support non aws buckets, and add default value. | 8.0.0 | | **0.3.1** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/4859))Add latency config parameter for aws-cloudwatch input | 8.0.0 | | **0.3.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/4158))Expose Default Region setting to UI | 8.0.0 | | **0.2.5** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/4103))Set default endpoint to empty string | 8.0.07.16.0 | | **0.2.4** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/3881))Fix proxy URL documentation rendering. | 8.0.07.16.0 | | **0.2.3** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/3610))Fix misspelling of Log Stream Prefix variable in manifest for aws-cloudwatch input | 8.0.07.16.0 | | **0.2.2** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/2819))update readme file | | | **0.2.1** | **Bug fix** ([View pull request](https://github.com/elastic/integrations/pull/2819))Add kibana version constraint | 8.0.07.16.0 | | **0.2.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/2710))Move s3 input and cloudwatch input into the same generic data stream | | | **0.1.0** | **Enhancement** ([View pull request](https://github.com/elastic/integrations/pull/2353))initial release | |