--- title: Troubleshooting Elastic Cloud Forwarder for AWS description: Resolve failed log forwarding, Lambda errors, and replay failed events in Elastic Cloud Forwarder for AWS. url: https://www.elastic.co/docs/reference/opentelemetry/edot-cloud-forwarder/aws/troubleshooting products: - Elastic Cloud Forwarder - Elastic Cloud Forwarder for AWS - Elastic Cloud Serverless - Elastic Observability applies_to: - Serverless Observability projects: Generally available - Elastic Cloud Hosted: Generally available - Self-managed Elastic deployments: Unavailable - Elastic Cloud Forwarder for AWS: Generally available since 1.0, Preview in 0.2 --- # Troubleshooting Elastic Cloud Forwarder for AWS This page helps you diagnose and resolve issues with Elastic Cloud Forwarder for AWS when logs are not being forwarded to Elasticsearch as expected. ## Key metrics to monitor Use **CloudWatch Metrics Explorer** to monitor your Elastic Cloud Forwarder Lambda function: | Metric | Expected behavior | |--------------------------|-------------------------------------------------------------------------------------------------------------------------------------| | **Duration** | S3: Increases with file size (2-5s typical). CloudWatch: Stays ~0.5s regardless of throughput. | | **ConcurrentExecutions** | Should not consistently hit the configured limit. CloudWatch sources use significantly higher concurrency than S3. | | **Errors** | Should be 0. | | **Throttles** | Should be 0. This is especially critical for CloudWatch sources, where throttling happens immediately if concurrency is undersized. | The `LambdaLogGroup` resource created by the CloudFormation stack stores all Lambda execution logs. Check these logs for processing errors, configuration issues, or data export failures. ## Lambda timeouts ### Symptoms Lambda execution times out before completing. Check file sizes and execution duration in CloudWatch metrics. ### Resolution The default 15-minute timeout handles all typical scenarios. For large files (multiple gigabytes), increase memory to allocate more CPU for faster processing. ## Concurrency throttling ### Symptoms - Ingestion lag despite fast individual executions. - `Throttles` metric showing non-zero values in CloudWatch. - `ConcurrentExecutions` metric consistently at the configured limit. ### Resolution Increase the `ElasticCloudForwarderConcurrentExecutions` parameter in your CloudFormation stack (named `EdotCloudForwarderConcurrentExecutions` on the legacy single log type templates). CloudWatch sources require higher concurrency than S3 sources for the same data rate. See [CloudWatch Logs sizing](/docs/reference/opentelemetry/edot-cloud-forwarder/aws/configure#cloudwatch-logs-sizing) for recommended values. ## High concurrency at low data rates (CloudWatch) ### Symptoms - Lambda concurrent executions appear disproportionately high relative to the data rate (for example, 20+ concurrent executions at under 1 MB/s). - No throttling or errors are present, but concurrency is much higher than you would expect from an equivalent S3 workload. ### Resolution This is expected behavior with CloudWatch subscriptions. CloudWatch delivers many small batches (~100-200 KB each) that each trigger a separate Lambda invocation. Concurrency is driven by `invocations/sec x avg_duration`, not by data volume alone. Unlike S3, where one large file is processed per invocation, CloudWatch sends many short-lived invocations in parallel. No action is needed unless you see throttling. If you do, increase `ElasticCloudForwarderConcurrentExecutions` (`EdotCloudForwarderConcurrentExecutions` on the legacy templates). ## A log type is not collected - Elastic Cloud Forwarder for AWS: Preview since 1.8 Applies to the unified templates, where one stack forwards multiple log types routed by S3 path pattern and CloudWatch log group or stream patterns. ### Symptoms - Objects arrive in a configured S3 bucket, but one log type produces no data in Elasticsearch, while other types from the same stack work. - CloudWatch logs from a configured Log Group produce no data, or land in the catch-all data stream (`logs-aws.cloudwatch.otel-default` in OpenTelemetry mode, `logs-aws.cloudwatch_logs-` in ECS mode) instead of the expected one. ### Resolution 1. Check the `EnabledLogTypes`, `WiredS3Buckets`, and `WiredLogGroups` outputs of the CloudFormation stack to confirm the type and its source are configured. If the type is missing, add it to `LogTypes` (or set `LogTypes` to `all`). 2. For S3 sources, confirm the objects match the type's path pattern. Types are matched by their standard AWS delivery prefix. If your logs are stored under a custom prefix, set it with `S3PatternOverrides`, for example `cloudtrail=my-org/audit/cloudtrail/*`. 3. Some log types have no standard delivery prefix and always require `S3PatternOverrides`: `s3access` (S3 server access logs) and the `netskope_*` types. 4. For CloudWatch sources, confirm the log group and stream names match the type's patterns, and adjust them with the log group and stream override parameters if needed. Log group overrides take precedence over stream overrides. Refer to [Unified template settings](/docs/reference/opentelemetry/edot-cloud-forwarder/aws/configure#unified-template-settings) for the override parameters. ## Failed log forwarding ### Symptoms You might experience one or more of the following symptoms: - Logs are not appearing in Elasticsearch or Kibana dashboards. - The S3 failure bucket contains unprocessed event files. - CloudWatch logs for the Lambda function show errors. - Lambda function metrics show increased error rates or throttling. - `StatusCode` errors when invoking the Lambda function. ### Resolution Open the AWS CloudWatch console and navigate to the `LambdaLogGroup` created by the CloudFormation stack. Look for error messages that indicate: - Network errors when connecting to the OTLP endpoint. - Authentication failures due to invalid or expired API key. - Log type mismatches between the source content and the configured log types (`LogTypes` on the unified templates, `EdotCloudForwarderS3LogsType` on the legacy templates). To get more detailed output, set `ElasticCloudForwarderLogLevel` (`EdotCloudForwarderLogLevel` on the legacy templates) to `debug` in your CloudFormation stack parameters and re-invoke the Lambda function. Confirm that your CloudFormation stack parameters are correct: - `OTLPEndpoint` points to a valid Managed OTLP endpoint. - `ElasticApiKey` is valid and not expired. - **Unified templates**: `LogTypes` includes the log types delivered to your sources, and the source lists (`S3Buckets`, `CloudWatchLogGroups`) contain the correct ARNs. Refer to [A log type is not collected](#a-log-type-is-not-collected) for routing mismatches. - **Legacy S3 sources**: `EdotCloudForwarderS3LogsType` matches the log format in your S3 bucket (`vpcflow`, `elbaccess`, `cloudtrail`, or `waf`). - **Legacy CloudWatch sources**: `EdotCloudForwarderCWLogType` matches the log type (`vpcflow` or `cloudtrail`) and `SourceCloudWatchLogGroupARN` points to an existing Log Group with the trailing `:*`. - The deployment region matches your log source region. In CloudWatch Metrics Explorer, review the Lambda function metrics: | Metric | What to look for | |------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `Errors` | Increased error count indicates processing failures. | | `Throttles` | High throttle count suggests you need to increase `ElasticCloudForwarderConcurrentExecutions` (`EdotCloudForwarderConcurrentExecutions` on the legacy templates). | | `Duration` | Long durations approaching the timeout may cause incomplete processing. | | `ConcurrentExecutions` | Compare against your reserved concurrency limit. | If events failed to process, they are stored in the S3 bucket specified by `S3FailureBucketARN`. Replay them by invoking the Lambda function with the `replayFailedEvents` trigger: ```sh aws lambda invoke \ --function-name \ --payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \ --cli-binary-format raw-in-base64-out /dev/null ``` Replace `` with the name of your Lambda function from the deployment.The following options are available: | Option | Description | Default | |-------------------|---------------------------------------------------------------------------------|---------| | `dryrun` | Run without processing events. Useful for understanding what would be replayed. | `false` | | `removeOnSuccess` | Remove the error event from the S3 failure bucket after successful processing. | `true` | Use `--timeout` with the AWS CLI to increase the Lambda timeout for custom invocations. If a timeout occurs, run the command multiple times to process all error events. If you're experiencing throttling or timeouts, consider adjusting the Lambda configuration. Refer to [Sizing and performance tuning](/docs/reference/opentelemetry/edot-cloud-forwarder/aws/configure#sizing-and-performance-tuning) for recommendations based on your log volume. ## Best practices - Monitor CloudWatch metrics regularly to catch issues early. - Set up CloudWatch alarms for Lambda errors and throttles. - Keep your API key up to date and rotate it before expiration. - Start with default sizing and increase concurrency or memory only when metrics indicate a need. - **S3 sources**: With the unified templates, verify the delivery prefix of each log type matches its path pattern, and set `S3PatternOverrides` for custom prefixes. With the legacy templates, ensure each log type uses a dedicated S3 bucket and CloudFormation stack. - **CloudWatch sources**: Start with 20 reserved concurrency and increase if the `Throttles` metric is non-zero. Memory of 256 MB is sufficient — increasing memory does not significantly improve CloudWatch processing performance. ## Resources - [Configure Elastic Cloud Forwarder for AWS](https://www.elastic.co/docs/reference/opentelemetry/edot-cloud-forwarder/aws/configure) - [Deploy Elastic Cloud Forwarder for AWS](https://www.elastic.co/docs/reference/opentelemetry/edot-cloud-forwarder/aws/deploy) - [Contact support](https://docs-demo.elastic.dev/docs/troubleshoot/ingest/opentelemetry/contact-support)