Nginx Integration


Version	3.2.2 (View all)
Subscription level What's this?	Basic
Developed by What's this?	Elastic
Ingestion method(s)	API, File
Minimum Kibana version(s)	9.0.0 8.13.0

The Nginx integration allows you to monitor Nginx servers. Time series index mode enabled for metrics data stream.

Use the Nginx integration to collect metrics and logs from your server. Then visualize that data in Kibana, use the Machine Learning app to find unusual activity in HTTP access logs, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.

For example, if you wanted to be notified if a certain number of client requests failed in a given time period, you could install the Nginx integration to send logs to Elastic. Then, you could view the logs stream into Elastic in real time in the Observability Logs app. You could also set up a new log threshold rule in the Logs app to alert you when there are more than a certain number of events with a failing status in a given time period.

Data streams

The Nginx integration collects two types of data: logs and metrics.

Logs help you keep a record of events that happen in your Nginx servers. This includes when a client request or error occurs.

Metrics give you insight into the state of your Nginx servers. This includes information like the total number of active client connections by status, the total number of client requests, and more.

Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Each data source was tested with a specific Nginx version. For more information see the Logs reference and Metrics reference.

Note

On Windows, the module was tested with Nginx installed from the Chocolatey repository.

Setup

For step-by-step instructions on how to set up an integration, see the Getting started guide.

Logs reference

Timezone support

This data source parses logs that don’t contain timezone information. For these logs, the Elastic Agent reads the local timezone and uses it when parsing to convert the timestamp to UTC. The timezone to be used for parsing is included in the event in the event.timezone field.

To disable this conversion, the event.timezone field can be removed using the drop_fields processor.

If logs originate from systems or applications with a timezone that is different than the local one, the event.timezone field can be overwritten with the original timezone using the add_fields processor.

Access Logs

Access logs collects the Nginx access logs.

Tested versions

The Nginx access logs stream was tested with Nginx 1.19.5 and 1.28.2.

Example

{
    "@timestamp": "2022-12-09T10:39:23.000Z",
    "_tmp": {},
    "agent": {
        "ephemeral_id": "34369a4a-4f24-4a39-9758-85fc2429d7e2",
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "data_stream": {
        "dataset": "nginx.access",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "created": "2022-12-09T10:39:38.896Z",
        "dataset": "nginx.access",
        "ingested": "2022-12-09T10:39:40Z",
        "kind": "event",
        "outcome": "success",
        "timezone": "+00:00",
        "type": [
            "access"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "66392b0697b84641af8006d87aeb89f1",
        "ip": [
            "172.18.0.7"
        ],
        "mac": [
            "02-42-AC-12-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.49-linuxkit",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.5 LTS (Focal Fossa)"
        }
    },
    "http": {
        "request": {
            "method": "GET"
        },
        "response": {
            "body": {
                "bytes": 97
            },
            "status_code": 200
        },
        "version": "1.1"
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/access.log"
        },
        "offset": 0
    },
    "nginx": {
        "access": {
            "remote_ip_list": [
                "127.0.0.1"
            ]
        }
    },
    "related": {
        "ip": [
            "127.0.0.1"
        ]
    },
    "source": {
        "address": "127.0.0.1",
        "ip": "127.0.0.1"
    },
    "tags": [
        "nginx-access"
    ],
    "url": {
        "original": "/server-status",
        "path": "/server-status"
    },
    "user_agent": {
        "device": {
            "name": "Other"
        },
        "name": "curl",
        "original": "curl/7.64.0",
        "version": "7.64.0"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

Field	Description	Type	Unit
@timestamp	Event timestamp.	date
cloud.image.id	Image ID for the cloud instance.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
input.type	Input type	keyword
log.offset	Log offset	long
nginx.access.remote_ip_list	An array of remote IP addresses. It is a list because it is common to include, besides the client IP address, IP addresses from headers like `X-Forwarded-For`. Real source IP is restored to `source.ip`.	keyword
nginx.access.response_time	Time to serve the request in milliseconds.	long	ms

Error Logs

Error logs collects the Nginx error logs.

Tested versions

The Nginx error logs stream was tested with Nginx 1.19.5 and 1.28.2.

Example

{
    "@timestamp": "2022-12-09T10:40:03.000Z",
    "agent": {
        "ephemeral_id": "34369a4a-4f24-4a39-9758-85fc2429d7e2",
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "data_stream": {
        "dataset": "nginx.error",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "created": "2022-12-09T10:40:18.973Z",
        "dataset": "nginx.error",
        "ingested": "2022-12-09T10:40:24Z",
        "kind": "event",
        "timezone": "+00:00",
        "type": [
            "error"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "66392b0697b84641af8006d87aeb89f1",
        "ip": [
            "172.18.0.7"
        ],
        "mac": [
            "02-42-AC-12-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.49-linuxkit",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.5 LTS (Focal Fossa)"
        }
    },
    "input": {
        "type": "log"
    },
    "log": {
        "file": {
            "path": "/tmp/service_logs/error.log"
        },
        "level": "warn",
        "offset": 0
    },
    "message": "conflicting server name \"localhost\" on 0.0.0.0:80, ignored",
    "nginx": {
        "error": {}
    },
    "process": {
        "pid": 1,
        "thread": {
            "id": 1
        }
    },
    "tags": [
        "nginx-error"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

Field	Description	Type
@timestamp	Event timestamp.	date
cloud.image.id	Image ID for the cloud instance.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
input.type	Input type	keyword
log.offset	Log offset	long
nginx.error.connection_id	Connection identifier.	long

Metrics reference

Stub Status Metrics

The Nginx stubstatus stream collects data from the Nginx ngx_http_stub_status module. It scrapes the server status data from the web page generated by ngx_http_stub_status. Please verify that your Nginx distribution comes with the mentioned module and it's enabled in the Nginx configuration file:

location /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

only allow requests from localhost
deny all other hosts

Replace 127.0.0.1 with your server’s IP address and make sure that this page accessible to only you.

Tested versions

The Nginx stubstatus stream was tested with Nginx 1.19.5 and 1.28.2. The Nginx integration metrics datastream is expected to work with all versions >= 1.19.

Example

{
    "@timestamp": "2024-02-08T08:12:10.668Z",
    "agent": {
        "ephemeral_id": "c3d516ba-b659-4190-a29d-d28200d74d48",
        "id": "2ea50bee-9250-43d1-8d70-949f242aa275",
        "name": "docker-fleet-agent",
        "type": "metricbeat",
        "version": "8.10.1"
    },
    "data_stream": {
        "dataset": "nginx.stubstatus",
        "namespace": "ep",
        "type": "metrics"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "2ea50bee-9250-43d1-8d70-949f242aa275",
        "snapshot": false,
        "version": "8.10.1"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "nginx.stubstatus",
        "duration": 1156104,
        "ingested": "2024-02-08T08:12:13Z",
        "module": "nginx"
    },
    "host": {
        "architecture": "x86_64",
        "containerized": false,
        "hostname": "docker-fleet-agent",
        "id": "28da52b32df94b50aff67dfb8f1be3d6",
        "ip": [
            "172.24.0.7"
        ],
        "mac": [
            "02-42-AC-18-00-07"
        ],
        "name": "docker-fleet-agent",
        "os": {
            "codename": "focal",
            "family": "debian",
            "kernel": "5.15.0-89-generic",
            "name": "Ubuntu",
            "platform": "ubuntu",
            "type": "linux",
            "version": "20.04.6 LTS (Focal Fossa)"
        }
    },
    "metricset": {
        "name": "stubstatus",
        "period": 10000
    },
    "nginx": {
        "stubstatus": {
            "accepts": 33,
            "active": 1,
            "current": 33,
            "dropped": 0,
            "handled": 33,
            "hostname": "elastic-package-service-nginx-1:80",
            "reading": 0,
            "requests": 33,
            "waiting": 0,
            "writing": 1
        }
    },
    "service": {
        "address": "http://elastic-package-service-nginx-1:80/server-status",
        "type": "nginx"
    },
    "tags": [
        "nginx-stubstatus"
    ]
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

Field	Description	Type	Metric Type
@timestamp	Event timestamp.	date
agent.id		keyword
cloud.account.id	The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.	keyword
cloud.availability_zone	Availability zone in which this host is running.	keyword
cloud.image.id	Image ID for the cloud instance.	keyword
cloud.instance.id	Instance ID of the host machine.	keyword
cloud.provider	Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.	keyword
cloud.region	Region in which this host is running.	keyword
container.id	Unique container id.	keyword
data_stream.dataset	Data stream dataset.	constant_keyword
data_stream.namespace	Data stream namespace.	constant_keyword
data_stream.type	Data stream type.	constant_keyword
event.dataset	Event dataset	constant_keyword
event.module	Event module	constant_keyword
host.containerized	If the host is a container.	boolean
host.name	Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.	keyword
host.os.build	OS build information.	keyword
host.os.codename	OS codename, if any.	keyword
nginx.stubstatus.accepts	The total number of accepted client connections.	long	counter
nginx.stubstatus.active	The current number of active client connections including Waiting connections.	long	gauge
nginx.stubstatus.current	The current number of client requests.	long	gauge
nginx.stubstatus.dropped	The total number of dropped client connections.	long	counter
nginx.stubstatus.handled	The total number of handled client connections.	long	counter
nginx.stubstatus.hostname	Nginx hostname.	keyword
nginx.stubstatus.reading	The current number of connections where Nginx is reading the request header.	long	gauge
nginx.stubstatus.requests	The total number of client requests.	long	counter
nginx.stubstatus.waiting	The current number of idle client connections waiting for a request.	long	gauge
nginx.stubstatus.writing	The current number of connections where Nginx is writing the response back to the client.	long	gauge
service.address	Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).	keyword

Anomaly Detection Configurations

These anomaly detection jobs are available in the Machine Learning app in Kibana when you have data that matches the query specified in the manifest.

Nginx access logs

Find unusual activity in HTTP access logs.

Job	Description
`visitor_rate_nginx`	HTTP Access Logs: Detect unusual visitor rates
`status_code_rate_nginx`	HTTP Access Logs: Detect unusual status code rates
`source_ip_url_count_nginx`	HTTP Access Logs: Detect unusual source IPs - high distinct count of URLs
`source_ip_request_rate_nginx`	HTTP Access Logs: Detect unusual source IPs - high request rates
`low_request_rate_nginx`	HTTP Access Logs: Detect low request rates

Screenshots

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.

Changelog

Version	Details	Minimum Kibana version
3.2.2	Enhancement (View pull request) Add group field to package manifest.	9.0.0 8.13.0
3.2.1	Enhancement (View pull request) Update package title and description to improve clarity of use.	9.0.0 8.13.0
3.2.0	Enhancement (View pull request) Add condition parameter support for logs and metrics inputs.	9.0.0 8.13.0
3.1.0	Enhancement (View pull request) Update dockerfile and readme for recent nginx version.	9.0.0 8.13.0
3.0.0	Breaking change (View pull request) Remove third-party pipeline for previously removed 'third-party REST API' input.	9.0.0 8.13.0
2.3.2	Bug fix (View pull request) Remove unused agent files.	9.0.0 8.13.0
2.3.1	Bug fix (View pull request) Fix Dashboard Links Panel by setting `useCurrentFilters` to `false`	9.0.0 8.13.0
2.3.0	Enhancement (View pull request) Use links panel in Dashboards.	9.0.0 8.13.0
2.2.1	Bug fix (View pull request) Update tests by using debian archive.	9.0.0 8.13.0
2.2.0	Enhancement (View pull request) Added optional parsing for request duration in access-log.	9.0.0 8.13.0
2.1.0	Enhancement (View pull request) Allow @custom pipeline access to event.original without setting preserve_original_event.	9.0.0 8.13.0
2.0.0	Enhancement (View pull request) Remove deprecated httpjson input.	9.0.0 8.13.0
1.25.1	Bug fix (View pull request) Added description to ssl nodes including links to documentation.	9.0.0 8.13.0
1.25.0	Enhancement (View pull request) Add support for Kibana `9.0.0`.	9.0.0 8.13.0
1.24.1	Bug fix (View pull request) Update links to getting started docs	8.13.0
1.24.0	Enhancement (View pull request) Deprecate third-party REST API import option.	8.13.0
1.23.0	Enhancement (View pull request) Add processor support for stubstatus data stream.	8.13.0
1.22.0	Enhancement (View pull request) ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.	8.13.0
1.21.0	Enhancement (View pull request) Add global filter on data_stream.dataset to improve performance.	8.12.0
1.20.0	Enhancement (View pull request) Enable secrets for sensitive fields. For more details, refer https://www.elastic.co/guide/en/fleet/current/agent-policy.html#agent-policy-secret-values	8.12.0
1.19.1	Bug fix (View pull request) Disable secrets for older stack versions due to errors.	8.8.0
1.19.0	Enhancement (View pull request) Add support for tags in stub status metrics	8.8.0
1.18.0	Enhancement (View pull request) Enable 'secret' for the sensitive fields, supported from 8.12.	8.8.0
1.17.0	Enhancement (View pull request) Limit request tracer log count to five.	8.8.0
1.16.0	Enhancement (View pull request) Update the package format_version to 3.0.0.	8.8.0
1.15.1	Bug fix (View pull request) Add null check and ignore_missing check to the rename processor	8.8.0
1.15.0	Enhancement (View pull request) Enable time series data streams for the metrics datasets. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html	8.8.0
1.14.1-beta	Bug fix (View pull request) Add beta flag	—
1.14.0	Enhancement (View pull request) Rename ownership from obs-service-integrations to obs-infraobs-integrations	8.8.0
1.13.0-beta	Enhancement (View pull request) Add a new flag to enable request tracing	—
1.12.0-beta	Enhancement (View pull request) Enable TSDB on Nginx for beta testing.	—
1.11.1	Bug fix (View pull request) Modifed the dimension field mapping to support public cloud deployment.	8.5.0
1.11.0	Enhancement (View pull request) Migrate "Access and error logs" dashboard visualizations to lens.	8.5.0
1.10.0	Enhancement (View pull request) Migrate "Logs Overview" dashboard visualizations to lens.	8.5.0
1.9.0	Enhancement (View pull request) Migrate "Metrics Overview" dashboard visualizations to lens.	8.5.0
1.8.0	Enhancement (View pull request) Added changes to support TSDB Enablement.	8.0.0
1.7.1	Enhancement (View pull request) Added categories and/or subcategories.	8.0.0
1.7.0	Enhancement (View pull request) Update ECS version to 8.5.1.	8.0.0
1.6.0	Enhancement (View pull request) Added infrastructure category.	8.0.0
1.5.1	Enhancement (View pull request) Remove unused visualizations	8.0.0
1.5.0	Enhancement (View pull request) Add ignore_older to log data streams	8.0.0
1.4.1	Enhancement (View pull request) Update documentation with additional context for new users.	8.0.0
1.4.0	Enhancement (View pull request) Migration of tile map to map object in dashboards	—
1.3.2	Enhancement (View pull request) Add documentation for multi-fields	—
1.3.1	Enhancement (View pull request) Add ML modules to readme	8.0.0 7.14.0
1.3.0	Enhancement (View pull request) Update to ECS 8.0	—
1.2.3	Bug fix (View pull request) Regenerate test files using the new GeoIP database	—
1.2.2	Bug fix (View pull request) Change test public IPs to the supported subset	—
1.2.1	Bug fix (View pull request) Fix ML module manifest query to ignore frozen and cold tiers	8.0.0 7.14.0
1.2.0	Enhancement (View pull request) Release nginx package for v8.0.0	8.0.0 7.14.0
1.1.2	Enhancement (View pull request) Uniform with guidelines	—
1.1.1	Bug fix (View pull request) Fix logic that checks for the 'forwarded' tag	—
1.1.0	Enhancement (View pull request) Update to ECS 1.12.0	7.14.0
1.0.0	Enhancement (View pull request) Release Nginx as GA	—
0.8.2	Enhancement (View pull request) Convert to generated ECS fields	—
0.8.1	Enhancement (View pull request) update to ECS 1.11.0	—
0.8.0	Enhancement (View pull request) Update integration description	—
0.7.0	Enhancement (View pull request) Set "event.module" and "event.dataset"	—
0.6.2	Enhancement (View pull request) Add support for Splunk authorization tokens	—
0.6.1	Bug fix (View pull request) Fix bug in Third Party REST API ingest pipeline	—
0.6.0	Enhancement (View pull request) update to ECS 1.10.0 and add event.original options	—
0.5.0	Enhancement (View pull request) Adds ML jobs for finding unusual activity in HTTP access logs	—
0.4.1	Enhancement (View pull request) update to ECS 1.9.0	—
0.3.11	Enhancement (View pull request) Updating package owner	—
0.3.10	Enhancement (View pull request) Fix compatibility with Kibana	—
0.1.0	Enhancement (View pull request) initial release	—