Fortinet FortiProxy Integration for Elastic

The Fortinet FortiProxy integration for Elastic lets you collect logs from your secure web gateway to monitor performance and security. It's designed to help you centralize data within the Elastic Stack for advanced visibility and threat detection. By ingesting these logs, you'll be able to monitor user behavior, investigate security events, and audit compliance across your network.

You can use this integration with the following:

• FortiProxy versions 7.x up to 7.4.3.
• Later versions of FortiProxy are expected to work correctly because they typically maintain backward compatibility with the default syslog format, but have not been tested.

You'll use the Elastic Agent to collect data from your FortiProxy device. You have several options for sending your syslog data to the agent:

• Use the tcp input for reliable, connection-oriented log delivery over the network.
• Use the udp input for high-performance transmission with minimal overhead.
• Use the filestream input if you want to read local log files from a specific path on your host.

Once the agent receives the logs through the log data stream, it sends them to your Elastic deployment. The data is parsed and mapped to the Elastic Common Schema (ECS), making it ready for use in dashboards and security analytics.

The Fortinet FortiProxy integration collects log messages of the following types:

• Traffic logs: Records of network traffic information including source and destination IP addresses, ports, protocols, bytes transferred, and session duration.
• HTTP transaction logs: Detailed HTTP/HTTPS request and response data including full URLs, methods, status codes, browser user agents, and timing details.
• UTM (Unified Threat Management) logs: Security-related logs generated by antivirus, web filtering, application control, data loss prevention (DLP), and SSL/SSH inspection modules.
• Event logs: System-level data including administrative logins, configuration changes, user authentication events, and system performance metrics.
• Security Rating logs: Security posture assessment results, including audit scores and compliance metrics generated by the FortiProxy security fabric.

Integrating Fortinet FortiProxy logs with Elastic provides you with enhanced security posture and operational visibility. You can use this integration for the following use cases:

• Security monitoring and threat detection: Use UTM and security rating logs to identify and mitigate threats like malware infections, unauthorized application usage, or security policy violations.
• Web traffic analysis: Analyze HTTP transaction logs to monitor web usage patterns, identify high-bandwidth sites, and ensure users follow acceptable use policies.
• Auditing and compliance: Track administrative logins, configuration changes, and user authentication events to meet regulatory requirements and maintain a searchable audit trail.
• Network performance troubleshooting: Use traffic logs to visualize flow patterns and correlate network data with other security and observability sources for faster incident response.

To use this integration, you need the following vendor-specific prerequisites:

• Root or super-admin level access to the FortiProxy CLI or web-based GUI to modify logging configurations.
• Network connectivity between the FortiProxy device and the Elastic Agent host. You must ensure that intermediate firewalls allow traffic on the configured port (the default is 514).
• The FortiProxy must be configured to use the default log format. This integration's parsing logic doesn't support custom or CSV formats.
• A valid license that enables logging features and the Security Fabric (if you're using Security Rating logs).

You also need the following Elastic prerequisites:

• Elastic Agent installed and enrolled in Fleet on a host that's network-accessible to the FortiProxy.
• Kibana and Elasticsearch version 8.0 or later for full compatibility with integration data streams.
• The Elastic Agent must be assigned to a policy that includes the Fortinet FortiProxy integration.
• The host running the Elastic Agent must listen on the specified port (for example, 514) and can't have local firewall rules (like iptables or Windows Firewall) blocking incoming syslog traffic.

You'll need to install Elastic Agent. For more details, check the Elastic Agent installation instructions. You can install only one Elastic Agent per host.

Elastic Agent is required to stream data from the syslog or log file receiver and ship the data to Elastic, where the events will then be processed using the integration's ingest pipelines.

Follow these steps to configure syslog collection using the command line interface:

1. Log in to the FortiProxy CLI using SSH or a console connection.
2. Enter the syslog configuration context:

   config log syslogd setting
   

3. Enable the syslog status and point it to your Elastic Agent's IP address (replace <Elastic_Agent_IP> with your actual value):

   set status enable
   set server <Elastic_Agent_IP>
   set port 514
   

4. Define the transport mode. Use set mode udp for standard delivery or set mode reliable for TCP delivery.
5. Ensure the format is compatible with the integration:

   set format default
   

6. Save the configuration and exit the context:

   end
   

7. (Optional) Configure which logs are sent by navigating to config log syslogd filter to set specific severity levels or event types.

Follow these steps to configure syslog collection using the web administration interface:

1. Log in to the FortiProxy web administration interface.
2. Navigate to Log & Report > Log Settings.
3. Locate the Remote Logging and Archiving section and toggle Send Logs to Syslog to the enabled position.
4. Enter the IP Address/FQDN of the host where the Elastic Agent is running.
5. Set the Port to 514 (or your chosen custom port).
6. Verify that CSV Format is disabled to ensure logs are sent in the required key-value format.
7. Click Apply to commit the changes.
8. Navigate to the different log type sections (Traffic, UTM, Event) in the GUI to ensure Local Log or Syslog is enabled for the specific events you wish to monitor.

You can find more information in these vendor resources:

• FortiProxy Administration Guide (Product Library)
• FortiProxy Administration Guide | Log settings

Follow these steps to set up the integration in Kibana:

1. In Kibana, navigate to Management > Integrations.
2. Search for Fortinet FortiProxy and select the integration.
3. Click Add Fortinet FortiProxy.
4. Configure the integration by selecting an input type and providing the necessary settings.

Choose the setup instructions that match your configuration. The input type should correspond to how you configured your FortiProxy to send logs in the vendor setup steps.

This input collects logs over a TCP socket. Configure these variables:

Under Advanced Options, you can configure the following:

• Tags: Custom tags to append to the events (for example, ['fortinet-fortiproxy', 'forwarded']).
• Processors: Add any custom processors to reduce fields or enhance metadata before the logs reach the ingest pipeline.
• SSL Configuration: Configure SSL options such as certificate and key if you're using encrypted transport.
• Custom TCP Options: Specify custom options such as framing: rfc6587. This is required if you're using set mode reliable on the FortiProxy device.

This input collects logs over a UDP socket. Configure these variables:

Under Advanced Options, you can configure the following:

• Tags: Custom tags to append to the events. Default: ['fortinet-fortiproxy', 'forwarded'].
• Custom UDP Options: Configure options such as read_buffer (e.g., 100MiB), max_message_size (e.g., 50KiB), and timeout (e.g., 300s).
• Processors: Add processors to filter or enhance data at the agent level.

This input collects logs directly from log files on the host where the Elastic Agent is running. Configure these variables:

Under Advanced Options, you can configure the following:

• Tags: Custom tags to append to the events. Default: ['fortinet-fortiproxy', 'forwarded'].
• Processors: Add processors for pre-parsing logic.

After you've configured the input, save and deploy the integration to an Elastic Agent policy.

Perform these actions to trigger data flow on your Fortinet FortiProxy device:

• Generate web traffic: From a client computer protected by the FortiProxy, browse to several different websites to generate traffic and HTTP transaction logs.
• Trigger security event: Attempt to access a known blocked URL category or test a non-malicious EICAR test file to trigger UTM antivirus or web filtering logs.
• Generate administrative event: Log out and log back into the FortiProxy management GUI or CLI to create an audit or system event log.

Follow these steps to check for data in Kibana:

1. Navigate to Analytics > Discover.
2. Select the logs-* data view.
3. Enter the KQL filter: data_stream.dataset : "fortinet_fortiproxy.log".
4. Verify logs appear and expand a log entry to confirm fields like event.dataset, source.ip, destination.ip, and event.action.
5. Navigate to Analytics > Dashboards and search for Fortinet FortiProxy to view the pre-built visualizations.

For help with Elastic ingest tools, check Common problems.

If you encounter issues while collecting logs from Fortinet FortiProxy, consider the following common scenarios:

• TCP framing issues: If you use TCP transport with reliable syslog mode, you must ensure that both the FortiProxy device and the integration settings use the same framing. Set framing: rfc6587 in the integration's Custom TCP Options to match the FortiProxy configuration and prevent parsing errors or merged events.
• Network port conflicts: If the Elastic Agent can't start the input, another service like a local syslog daemon might be using the configured port. You can check for port usage on Linux systems using the command netstat -tulpn.
• Firewall or connectivity problems: If logs aren't appearing in Kibana, verify that the FortiProxy device can reach the Elastic Agent host. You can use a packet capture tool like tcpdump port 514 on the Agent host to confirm that traffic is reaching the network interface.
• Incorrect log format: Ensure that your FortiProxy device is configured to use the default log format with the CLI command set format default. If the device sends logs in CSV or a custom format, the integration's ingest pipeline won't be able to parse the data correctly.
• Incomplete log messages or truncation: If very large log entries, such as those from SSL inspection, are appearing cut off, you might need to increase the max_message_size in the integration's advanced settings for the TCP or UDP input.
• Troubleshooting parsing failures: You can look for the error.message field in Kibana to identify why a specific log failed to parse. To help with this, you should enable the Preserve original event option in the integration settings so you can compare the raw log in event.original against the expected format.

For more information on architectures that can be used for scaling this integration, check the Ingest Architectures documentation.

To ensure optimal performance in high-volume environments, consider these recommendations:

• If you have high-reliability requirements, use TCP transport (mode reliable) to ensure you don't lose logs during network congestion.
• When you use TCP, make sure rfc6587 framing is enabled in the configuration to handle message boundaries correctly.
• For high-performance environments where you prioritize low overhead, UDP transport is recommended, though it doesn't guarantee delivery.
• You can manage data volume by configuring your FortiProxy appliance to forward only necessary events, which reduces the load on the ingest pipeline.
• Use the FortiProxy CLI command config log syslogd filter to exclude low-priority traffic logs or limit logging to specific severity levels, like warning and above. This minimizes processing overhead on both the FortiProxy device and the Elastic Agent.
• For high-throughput environments with thousands of concurrent web sessions, you should deploy multiple Elastic Agents behind a network load balancer to distribute traffic evenly.
• Place your Agents geographically close to the data source to minimize latency and ensure the host machine has enough CPU and memory to parse complex UTM logs.

These inputs can be used with this integration:

ssl.enabled: true
ssl.certificate: "/path/to/server.crt"
ssl.key: "/path/to/server.key"
ssl.certificate_authorities: ["/path/to/ca.crt"]
ssl.client_authentication: "optional"

You can find more information about FortiProxy logging and configuration in the following resources:

• Fortinet Documentation Library
• Fortinet FortiProxy Documentation Library
• FortiProxy Log Message Reference
• FortiProxy Administration Guide | Log settings

The log data stream provides events from Fortinet FortiProxy appliances. This integration collects various log types to provide visibility into network traffic, security threats, and system activities.

The following types of data are collected in this data stream:

• Traffic logs that include network information such as source and destination IP addresses, ports, and protocols.
• HTTP transaction logs that contain detailed request and response data, including URLs, methods, and status codes.
• Unified Threat Management (UTM) logs generated by security modules like antivirus, web filtering, application control, and DLP.
• Event logs that capture system-level data, administrative logins, and configuration changes.
• Security Rating logs that provide assessment results and compliance metrics from the security fabric.

{
    "@timestamp": "2024-04-11T02:56:17.000Z",
    "agent": {
        "ephemeral_id": "5bdc4789-78f8-49b0-807e-5a6c1e876d58",
        "id": "5b7ea00b-603f-4de7-b7f7-240330ab7d50",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.2"
    },
    "client": {
        "bytes": 798,
        "ip": "10.0.0.3",
        "nat": {
            "ip": "10.0.128.2",
            "port": 53184
        },
        "port": 47886
    },
    "data_stream": {
        "dataset": "fortinet_fortiproxy.log",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "bytes": 125800732,
        "geo": {
            "continent_name": "Asia",
            "country_iso_code": "BT",
            "country_name": "Bhutan",
            "location": {
                "lat": 27.5,
                "lon": 90.5
            }
        },
        "ip": "67.43.156.10",
        "port": 443
    },
    "ecs": {
        "version": "8.17.0"
    },
    "elastic_agent": {
        "id": "5b7ea00b-603f-4de7-b7f7-240330ab7d50",
        "snapshot": false,
        "version": "8.13.2"
    },
    "event": {
        "action": "accept",
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "code": "0000000010",
        "dataset": "fortinet_fortiproxy.log",
        "duration": 8089000000000,
        "ingested": "2024-06-07T14:49:44Z",
        "kind": "event",
        "original": "<189>date=2024-04-10 time=19:56:17 devname=\"TEST-PXY01\" devid=\"FPXTESTPXY01\" eventtime=1712771778239212440 tz=\"-0700\" logid=\"0000000010\" type=\"traffic\" subtype=\"forward\" level=\"notice\" vd=\"root\" srcip=10.0.0.3 srcport=47886 srcintf=\"port2\" srcintfrole=\"lan\" dstcountry=\"United States\" srccountry=\"Reserved\" dstip=67.43.156.10 dstport=443 dstintf=\"port1\" dstintfrole=\"undefined\" sessionid=1781818019 service=\"HTTPS\" proxyapptype=\"web-proxy\" proto=6 action=\"accept\" policyid=1 policytype=\"proxy-policy\" poluuid=\"27b09930-033d-51ef-0c72-6c1221a8d893\" policyname=\"test-proxy\" trandisp=\"snat\" transip=10.0.128.2 transport=53184 clientip=10.0.0.3 duration=8089 wanin=125800732 rcvdbyte=125800732 wanout=632 lanin=798 sentbyte=798 lanout=125824455 appcat=\"unscanned\" utmaction=\"allow\"",
        "start": "2024-04-10T17:56:18.239Z",
        "timezone": "-0700"
    },
    "fortinet": {
        "proxy": {
            "dstintfrole": "undefined",
            "lanin": 798,
            "lanout": 125824455,
            "proxyapptype": "web-proxy",
            "sessionid": "1781818019",
            "srcintfrole": "lan",
            "subtype": "forward",
            "trandisp": "snat",
            "type": "traffic",
            "utmaction": "allow",
            "vd": "root",
            "wanin": 125800732,
            "wanout": 632
        }
    },
    "input": {
        "type": "filestream"
    },
    "log": {
        "file": {
            "device_id": "35",
            "inode": "80",
            "path": "/tmp/service_logs/fortinet-fortiproxy.log"
        },
        "level": "notice",
        "offset": 15140,
        "syslog": {
            "facility": {
                "code": 23
            },
            "priority": 189,
            "severity": {
                "code": 5
            }
        }
    },
    "network": {
        "bytes": 125801530,
        "iana_number": "6",
        "protocol": "https",
        "transport": "tcp"
    },
    "observer": {
        "egress": {
            "interface": {
                "name": "port1"
            }
        },
        "hostname": "TEST-PXY01",
        "ingress": {
            "interface": {
                "name": "port2"
            }
        },
        "product": "FortiProxy",
        "serial_number": "FPXTESTPXY01",
        "type": "proxy",
        "vendor": "Fortinet"
    },
    "rule": {
        "category": "unscanned",
        "id": "1",
        "name": "test-proxy",
        "ruleset": "proxy-policy",
        "uuid": "27b09930-033d-51ef-0c72-6c1221a8d893"
    },
    "server": {
        "bytes": 125800732,
        "geo": {
            "continent_name": "Asia",
            "country_iso_code": "BT",
            "country_name": "Bhutan",
            "location": {
                "lat": 27.5,
                "lon": 90.5
            }
        },
        "ip": "67.43.156.10",
        "port": 443
    },
    "source": {
        "bytes": 798,
        "ip": "10.0.0.3",
        "nat": {
            "ip": "10.0.128.2",
            "port": 53184
        },
        "port": 47886
    },
    "tags": [
        "preserve_original_event",
        "fortinet-fortiproxy",
        "forwarded"
    ]
}

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.