Machine learning in Kibana

As data sets increase in size and complexity, the human effort required to inspect dashboards or maintain rules for spotting infrastructure problems, cyber attacks, or business issues becomes impractical. Elastic machine learning features such as anomaly detection and outlier detection make it easier to notice suspicious activities with minimal human interference.

Kibana includes a free Data Visualizer that helps you understand your data's structure, distribution, and quality before you act on it. If your data is stored in Elasticsearch and contains a time field, the Data Visualizer also helps you identify possible fields for anomaly detection.

Tip

Click Zoom in or Zoom out next to the date picker to quickly narrow or widen the time range when exploring your data.

Data Visualizer for sample web logs data

You can upload different file formats for analysis with the Data Visualizer.

File formats supported up to 500 MB:

CSV
TSV
NDJSON
Log files

File formats supported up to 60 MB:

PDF
Microsoft Office files (Word, Excel, PowerPoint)
Plain text (TXT)
Rich text (RTF)
Open Document Format (ODF)

The Data Visualizer identifies the file format and field mappings, and you can import the data into an Elasticsearch index. To upload files, refer to Upload data files. To change the default file size limit, refer to fileUpload:maxFileSize in Advanced Settings.

If Elastic Stack security features are enabled, users must have the necessary privileges to use machine learning features. Refer to Set up machine learning features.

Note

There are limitations in machine learning features that affect Kibana. For more information, refer to Machine learning.

Data drift

You can find the Data drift view in Machine Learning → Data Visualizer in Kibana or by using the global search field. The Data drift view shows you the differences in each field for two different time ranges in a given data view. The view helps you to visualize the changes in your data over time and enables you to understand its behavior better.

You select a data view to analyze, along with separate time ranges for the reference and comparison data in the histogram chart, adjusting each range by moving its brush. Run analysis then compares the reference and comparison periods.

You can decide whether you want to view all the fields in the data view or only the ones that contain drifted data. The analysis results table displays the fields, their types, if drift is detected, the p-value that indicates how significant the detected change is, the reference and comparison distribution, and the comparison chart. To expand the results for a particular field, select its row's arrow icon.