Cilium Tetragon Integration for Elastic

The Cilium Tetragon integration for Elastic enables you to monitor and analyze events from Tetragon, a Kubernetes-aware security observability and runtime enforcement tool supported by the Cloud Native Computing Foundation (CNCF). This integration provides granular visibility into the internal workings of your Kubernetes clusters by collecting security event logs, allowing you to visualize data in Kibana, set up alerts, and respond to security events in real time.

This integration is compatible with Cilium Tetragon running in Kubernetes environments. It has been specifically tested and validated using a sidecar deployment pattern.

This integration works by collecting JSON-formatted event logs generated by the Cilium Tetragon agent. You typically deploy a Filebeat sidecar container that reads these logs from a shared volume and forwards them to your Elastic deployment. The integration then processes and indexes the logs through the log data stream, making them available for analysis in Kibana. This approach leverages eBPF technology to provide deep runtime security observability and enforcement without requiring changes to your application code.

The Cilium Tetragon integration collects runtime security events and system-level telemetry from your Kubernetes clusters using eBPF-based monitoring.

The integration collects log messages of the following types in the log data stream:

• Security event logs: Comprehensive runtime security events from the Tetragon agent.
• Process metadata: Detailed information regarding process execution, including binary paths, PIDs, and parent-child relationships.
• Network logs: Connectivity data including source/destination IPs, ports, and protocol-specific details for cluster traffic.
• System call information: Detailed logs of system calls monitored by Tetragon's eBPF-based enforcement engine.

Integrating Cilium Tetragon logs with Elastic provides visibility and security monitoring for containerized environments. You can use this integration for the following use cases:

• Kubernetes security observability: Monitor and analyze security events from applications running in Kubernetes to gain visibility into process executions and kernel-level activities.
• Runtime threat detection: Detect and respond to security threats by analyzing process behavior, file access patterns, and system call activities.
• Compliance and audit logging: Capture detailed audit trails of process lifecycle events and system activities with full Kubernetes context for forensic analysis and regulatory requirements.
• Container security monitoring: Track process lifecycle events within containers and correlate security events with Kubernetes workload types, namespaces, and labels.

To use this integration, you'll need the following environment and vendor-specific components:

• A running Kubernetes cluster with nodes that support eBPF.
• Helm v3 installed and configured locally to manage the Tetragon deployment.
• kubectl access with cluster-admin or equivalent permissions to create Role-Based Access Control (RBAC) resources, DaemonSets, and ConfigMaps in the kube-system namespace.
• Permission to run containers with privileged: true or specific capabilities like CAP_SYS_ADMIN, and the sidecar collector with runAsUser: 0.

You also need the following Elastic Stack components:

• Elastic Stack version 8.13.0 or later.
• The Cilium Tetragon integration assets installed in Kibana.
• Elasticsearch credentials with permissions to create and write to indices.
• Outbound network connectivity from Kubernetes nodes to the Elasticsearch cluster, typically on port 443 or 9200.

This integration uses a Filebeat sidecar approach to collect logs from Kubernetes environments and send them to Elastic using a cloud-native design.

Elastic Agent is not used to collect data for this integration. Instead, you'll install the integration assets in Kibana to provide the necessary ingest pipelines and dashboards, then configure Tetragon to export logs to Elasticsearch using a Filebeat sidecar.

To set up Cilium Tetragon to send data to Elastic, you'll need to configure a Filebeat sidecar and update your Tetragon deployment. Follow these steps to configure the system:

1. Create a Filebeat ConfigMap to define the log processing logic. Save the following content as filebeat-config.yaml, ensuring you update the connection details:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: filebeat-configmap
     namespace: kube-system
   data:
     filebeat.yml: |
       filebeat.inputs:
         - type: filestream
           id: tetragon-log
           enabled: true
           paths:
             - /var/run/cilium/tetragon/*.log
       path.data: /usr/share/filebeat/data
       processors:
         - timestamp:
             field: "time"
             layouts:
               - '2006-01-02T15:04:05Z'
               - '2006-01-02T15:04:05.999Z'
               - '2006-01-02T15:04:05.999-07:00'
             test:
               - '2019-06-22T16:33:51Z'
               - '2019-11-18T04:59:51.123Z'
               - '2020-08-03T07:10:20.123456+02:00'
       setup.template.name: logs
       setup.template.pattern: "logs-cilium_tetragon.*"
       output.elasticsearch:
         hosts: ["https://<<elasticsearch_host>> (replace with your actual value)"]
         username: "<<elasticsearch_username>> (replace with your actual value)"
         password: "<<elasticsearch_password>> (replace with your actual value)"
         index: logs-cilium_tetragon.log-default
   

2. Apply the configuration to your cluster:

   kubectl apply -f filebeat-config.yaml
   

3. Create a filebeat-helm-values.yaml file to enable JSON export and inject the sidecar container into the Tetragon pods, replacing the image with your stack version tag:

   export:
     securityContext:
       runAsUser: 0
       runAsGroup: 0
     stdout:
       enabledCommand: false
       enabledArgs: false
       image:
         override: "docker.elastic.co/beats/filebeat:<<stack version>>"
       extraVolumeMounts:
         - name: filebeat-config
           mountPath: /usr/share/filebeat/filebeat.yml
           subPath: filebeat.yml
         - name: filebeat-data
           mountPath: /usr/share/filebeat/data
   extraVolumes:
     - name: filebeat-data
       hostPath:
         path: /var/run/cilium/tetragon/filebeat
         type: DirectoryOrCreate
     - name: filebeat-config
       configMap:
         name: filebeat-configmap
         items:
           - key: filebeat.yml
             path: filebeat.yml
   

4. Deploy or update Tetragon using Helm with your override file:

   helm repo add cilium https://helm.cilium.io
   helm repo update
   helm install tetragon -f filebeat-helm-values.yaml cilium/tetragon -n kube-system
   

For more information on configuring Tetragon, refer to the following resources:

• Tetragon | Deploy on Kubernetes
• Tetragon Helm Chart Reference

To install the necessary assets for the Cilium Tetragon integration, follow these steps:

1. In Kibana, navigate to Management > Integrations.
2. Search for and select Cilium Tetragon.
3. Click Add Cilium Tetragon.
4. Follow the prompts to install the integration assets. You must select Add Integration Only to install dashboards and pipelines without enrolling an Elastic Agent, as this integration uses the Filebeat sidecar.
5. Configure the general integration settings:
   • Integration name: A unique name for this integration instance (e.g., tetragon-kubernetes).
   • Namespace: The namespace where data will be indexed (e.g., default).
6. Click Save and continue.

To verify that the integration is working properly and data is flowing into Elasticsearch, follow these steps:

1. Check that the Tetragon pods are running in the kube-system namespace:

   kubectl get pods -n kube-system -l app.kubernetes.io/name=tetragon
   

2. Trigger a Tetragon event by running a command inside any pod in your cluster which will trigger a Tetragon event. The actions required will depend on your configured Tetragon policies.
3. In Kibana, navigate to Analytics > Discover.
4. Select the logs-* data view and enter the following KQL filter to verify logs appear: data_stream.dataset : "cilium_tetragon.log".
5. Confirm that events contain expected fields such as event.dataset and event.action.
6. Navigate to Analytics > Dashboards and search for "Cilium Tetragon" to view the pre-built visualizations.

For help with Elastic ingest tools, check Common problems.

This integration involves several components within your Kubernetes cluster. If you encounter issues, review these common scenarios:

• No data appearing in Elasticsearch after installation:
  • Verify that the Filebeat ConfigMap was created successfully using kubectl get configmap -n kube-system filebeat-configmap.
  • Check Tetragon pod logs for errors using kubectl logs -n kube-system <tetragon-pod-name>.
  • Verify the Filebeat sidecar is running and exporting logs using kubectl logs -n kube-system <tetragon-pod-name> -c export-stdout.
  • Confirm Elasticsearch credentials and host configuration are correct in your ConfigMap.
  • Test network connectivity from within the Kubernetes cluster to your Elasticsearch endpoint.
• Tetragon pods not starting or in CrashLoopBackOff state:
  • Check pod events for security context or resource issues using kubectl describe pod -n kube-system <tetragon-pod-name>.
  • Verify the node kernel version supports eBPF. Use uname -r to check (kernel 4.9+ is recommended, while 5.3+ is required for full feature support).
• Volume mount mismatch:
  • Ensure the exportDirectory in the Tetragon configuration matches the mountPath configured in both the Tetragon container and the Filebeat sidecar container.
  • If these paths don't align, Filebeat will attempt to read from an empty or non-existent directory.
• Missing or incorrect Kubernetes metadata:
  • Verify that Tetragon has the proper RBAC permissions to access the Kubernetes API.
  • Check if Tetragon is configured with the correct cluster name by verifying the --cluster-name flag in your Helm values.
• RBAC and permissions issues:
  • If the Filebeat sidecar fails to start, verify the securityContext is set to runAsUser: 0. Filebeat often requires root permissions to read logs from host-mounted or shared volumes in Kubernetes.
• Only some events are being collected:
  • Check the tetragon.exportAllowList values in your deployment to ensure the desired event types (such as PROCESS_EXEC, PROCESS_EXIT, or PROCESS_KPROBE) are included.

For further assistance with Cilium Tetragon, refer to the following resources:

• Tetragon Troubleshooting Guide

For more information on architectures that can be used for scaling this integration, check the Ingest Architectures documentation.

To ensure optimal performance in high-volume Kubernetes environments, consider the following configuration strategies:

• Transport and collection strategy: This integration uses the filestream input via a Filebeat sidecar. By reading logs directly from a shared volume within the pod, you'll ensure low-latency access and avoid the network overhead of external syslog collection. This sidecar approach allows log collection to scale linearly as you add more Cilium Tetragon pods to your cluster.
• Data volume management: You can manage high volumes of security data by configuring the tetragon.exportAllowList and tetragon.exportDenyList settings. Filtering events at the source (the Tetragon engine) helps you avoid ingesting unnecessary system events, which reduces disk I/O and minimizes the processing load on your Elasticsearch cluster.

The following inputs are supported by this integration:

You can find more information about Cilium Tetragon in the following resources:

• Official Tetragon Documentation
• Tetragon Concepts: Events & JSON Export

The Cilium Tetragon integration includes the following data stream:

The log data stream captures security and observability events from Tetragon, such as process executions, file operations, and network activity.

This is a list of the fields exported by this data stream: