Domain Generation Algorithm Detection

The Domain Generation Algorithm (DGA) Detection package contains assets to detect DGA activity in your network data. This package requires a Platinum subscription. Please ensure that you have a Trial or Platinum level subscription installed on your cluster before proceeding. This package is licensed under Elastic License 2.0.

This package supports data from Elastic Endpoint via Elastic Defend or Packetbeat on Linux, macOS, and Windows, although Elastic Defend is recommended. Prior to using this integration, Elastic Defend should be installed through Elastic Agent (or Packetbeat should be enrolled) and collecting data from hosts. See Configure endpoint protection with Elastic Defend for more information.

The following blogs provide additional context. For the most current installation instructions, always follow the steps in this guide.

• Detect domain generation algorithm (DGA) activity with new Kibana integration
• Combining supervised and unsupervised machine learning for DGA detection

1. Upgrading: If upgrading from a version below v3.0.0, see the section v3.0.0 and beyond.
2. Add the Integration Package: Install the package via Management > Integrations > Add Domain Generation Algorithm Detection. Configure the integration name and agent policy. Click Save and Continue. (Note that this integration does not rely on an agent, and can be assigned to a policy without an agent.)
3. Install assets: Install the assets by clicking Settings > Install Domain Generation Algorithm Detection assets.
4. Configure the pipeline: To configure the pipeline you can use one of the following steps:
   • If using Elastic Defend, add a custom pipeline to the data stream. Go to Stack Management > Ingest Pipelines, and check if the pipeline logs-endpoint.events.network@custom exists. If it does not exist, you can create it by running the following command in the Dev Console. Be sure to replace <VERSION> with the current package version.

     PUT _ingest/pipeline/logs-endpoint.events.network@custom
     {
       "processors": [
         {
           "pipeline": {
             "name": "<VERSION>-ml_dga_ingest_pipeline",
             "ignore_missing_pipeline": true,
             "ignore_failure": true
           }
         }
       ]
     }
     

   • If logs-endpoint.events.network@custom already exists, select the three dots next to it and choose Edit. Click Add a processor. Select Pipeline for Processor, enter <VERSION>-ml_dga_ingest_pipeline for name (replacing <VERSION> with the current package version), and check Ignore missing pipeline and Ignore failures for this processor. Select Add Processor.
   • If using an Elastic Beat such as Packetbeat, add the ingest pipeline to it by adding a simple configuration setting to packetbeat.yml and skip to the Add preconfigured anomaly detection jobs section in these instructions.
5. Add the required mappings to the index or component template: Go to Stack Management > Index Management > Component Templates. Templates that can be edited to add custom components will be marked with a @custom suffix. For instance, the custom component template for Elastic Defend network events is logs-endpoint.events.network@custom. Note: Do not attempt to edit the @package template.
   • If the @custom component template does not exist, you can execute the following command in the Dev Console to create it and then continue to the Rollover section in these instructions.

     PUT _component_template/{COMPONENT_TEMPLATE_NAME}@custom
     {
       "template": {
         "mappings": {
           "properties": {
             "ml_is_dga": {
               "type": "object",
               "properties": {
                 "malicious_prediction": {
                   "type": "long"
                 },
                 "malicious_probability": {
                   "type": "float"
                 }
               }
             }
           }
         }
       }
     }
     

   • If the @custom component template already exists, you will need to edit it to add mappings for data to be properly enriched. Click the three dots next to it and select Edit.
   • Proceed to the mappings step in the UI. Click Add Field at the bottom of the page and create an an Object field for ml_is_dga.
   • Finally create two properties under ml_is_dga.
   • The first for malicious_prediction of type Long and then for malicious_probability or type Float.
   • Your component mappings should look like the following:
   • Click Review then Save Component Template.
6. Rollover Depending on your environment, you may need to rollover in order for these mappings to get picked up. The deault index pattern for Elastic Defend is logs-endpoint.events.network-default.

   POST INDEX_NAME/_rollover
   

7. (Optional) Create a data view for your network logs.
8. Add preconfigured anomaly detection jobs: In Stack Management -> Anomaly Detection Jobs, you will see Select data view or saved search. Select the data view created in the previous step. Then under Use preconfigured jobs you will see DGA. When you select the card, you will see a pre-configured anomaly detection job that you can create. Note this job is only useful for indices that have been enriched by the ingest pipeline.

You can also enable detection rules to alert on DGA activity in your environment, based on anomalies flagged by the above ML jobs. As of version 2.0.0 of this package, these rules are available as part of the Detection Engine in Security > Rules, and can be found using the tag Use Case: Domain Generated Algorithm Detection. See this documentation for more information on importing and enabling the rules. Warning: if the ingest pipeline hasn't run for some reason, such as no eligible data has come in yet, or the required mapping has not been added, you won't be able to see this card yet. If that is the case, try troubleshooting the ingest pipeline, and if any predictions have been populated yet.

In Security > Rules, filtering with the “Use Case: Domain Generation Algorithm Detection” tag

To customize the datafeed query and other settings such as model memory limit, frequency, query delay, bucket span and influencers for the Domain Generation Algorithm Detection ML jobs, follow the steps below.

1. To update the datafeed query, stop the datafeed and select Edit job from the Actions menu.
2. In the Edit job window, navigate to the Datafeed section and update the query filters. You can add or remove field values to help reduce noise and false positives based on your environment.
3. You may also update the model memory limit if your environment has high data volume or if the job requires additional resources. Go to the Job details section and update the Model memory limit and hit Save. For more information on resizing ML jobs, refer to the documentation.
4. In order to do more advanced changes to your job, clone the job by selecting Clone job from the Actions menu.
5. In the cloned job, you can update datafeed settings such as Frequency and Query delay, which help control how often data is analyzed and account for ingestion delays.
6. You can also modify the job configuration by adjusting the Bucket span and by adding or removing Influencers to improve anomaly attribution.
7. Finally, assign a new Job ID, and click on Create job, and start the datafeed to apply the updated settings.

v3.0.0 of this package requires Elastic Stack version 9.4 or later. It introduces support for Entity Analytics (EA), adding new fields for proper entity resolution.

• This package installs new ML jobs which include _ea suffix in their names, as outlined below. New detection rules are also included.
• Previously installed ML jobs and rules will continue to run, allowing time to transition to the new Entity Analytics assets.
• Important: We recommend installing the new ML jobs and verifying that they are properly set up, collecting data, and generating anomalies before deleting the old jobs and upgrading to the new version of the detection rules available in 9.4. The new detection rules reference ML job IDs with the _ea suffix and are not compatible with older versions of the jobs.

The new Entity Analytics ML job IDs are:

• dga_high_sum_probability_ea

After confirming the new Entity Analytics ML jobs are running correctly, you can remove the following deprecated assets that have been superseded by the new Entity Analytics versions (Elastic stack 9.4+):

• Delete old ML jobs: Navigate to Stack Management -> Anomaly Detection Jobs and delete the following jobs:
  • dga_high_sum_probability

v2.0.0 of the package introduces breaking changes, namely deprecating detection rules from the package. To continue receiving updates to DGA Detection, we recommend upgrading to v2.0.0 after doing the following:

• Uninstall existing rules associated with this package: Navigate to Security > Rules and delete the following rules:
  • Machine Learning Detected DGA activity using a known SUNBURST DNS domain
  • Machine Learning Detected a DNS Request Predicted to be a DGA Domain
  • Potential DGA Activity
  • Machine Learning Detected a DNS Request With a High DGA Probability Score

Depending on the version of the package you're using, you might also be able to search for the above rules using the tag DGA

• Upgrade the DGA package to v2.0.0 using the steps here
• Install the new rules as described in the Enable detection rules section below

Usage in production requires that you have a license key that permits use of machine learning features.

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.