Collective Intelligence Framework v3 Integration
Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability This functionality is generally available and ready for production usage. Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud. Availability This functionality is generally available and ready for production usage. The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats. Availability This functionality is generally available and ready for production usage. Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments. This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.
Serverless Observability
Generally available
Serverless Security
Generally available
Stack
9.1+
Generally available since 9.1
| Version | 1.25.0 (View all) |
| Managed integration release status | GA |
| Subscription level What's this? |
Basic |
| Developed by What's this? |
Community |
| Ingestion method(s) | API |
| Minimum Kibana version(s) | 9.1.0 8.19.0 |
You can use this integration as an Elastic Managed integration on Elastic Cloud Hosted deployments running this version or later.
This integration connects with the REST API from the running CIFv3 instance to retrieve indicators.
Elastic Managed integrations are only supported on Elastic Cloud Serverless and Elastic Cloud Hosted deployments. An Elastic Managed integration lets you ingest data from a cloud source while avoiding the orchestration, management, and maintenance associated with standard ingest infrastructure. Elastic runs the collector for you, so you can focus on your data instead of the infrastructure that collects it.
For more information, refer to Elastic Managed integrations and the Elastic Managed integrations FAQ.
Indicators are expired after a certain duration. An Elastic Transform is created for a source index to allow only active indicators to be available to the end users. The transform creates a destination index named logs-ti_cif3_latest.dest_feed* which only contains active and unexpired indicators. Destination indices are aliased to logs-ti_cif3_latest.feed. The indicator match rules and dashboards are updated to show only active indicators.
| Indicator Type | Indicator Expiration Duration |
|---|---|
ipv4-addr |
45d |
ipv6-addr |
45d |
domain-name |
90d |
url |
365d |
file |
365d |
| All Other Types | Derived from IOC Expiration Duration setting |
Threat indicators are re-collected across polling intervals, and the latest transform keeps the active, deduplicated view in its destination index. The source data streams therefore hold repeated copies of the same indicators.
The package bounds the growth of these source data streams with a retention that depends on the deployment type:
| Data stream | Self-managed and Elastic Cloud Hosted (ILM policy) | Serverless (data stream lifecycle) |
|---|---|---|
logs-ti_cif3.feed-* |
logs-ti_cif3.feed-default_policy: roll over after 2d, delete 3d after rollover |
delete 5d after rollover |
On self-managed and Elastic Cloud Hosted deployments the ILM policy applies. The data stream lifecycle shipped with the package is not used there. ILM counts the delete age from the rollover of a backing index, so a document can remain for up to the rollover age plus the delete age. On Serverless, ILM is not available and the data stream lifecycle applies instead. It also works per backing index: Elasticsearch rolls the write index over automatically on age, size, or document count, and deletes a backing index once the retention has passed since it rolled over. A document therefore stays for the retention plus up to one rollover interval. The rollover age is derived from the retention and is an implementation detail that Elasticsearch may change. Where the package installs a transform, the transform's destination indices are not affected by either.
To keep data for a different period:
- Self-managed and Elastic Cloud Hosted: edit the ILM policy in Kibana under Stack Management → Index Lifecycle Policies, or with
PUT _ilm/policy/<policy name>. A package upgrade reinstalls the package's ILM policies, so check your change after upgrading. - Serverless: set the retention on the data stream, for example
PUT _data_stream/logs-ti_cif3.feed-default/_lifecyclewith the body{"data_retention": "90d"}. Replacedefaultwith your namespace.
The CIFv3 integration collects threat indicators based on user-defined configuration including a polling interval, how far back in time it should look, and other filters like indicator type and tags.
CIFv3 confidence field values (0..10) are converted to ECS confidence (None, Low, Medium, High) in the following way:
| CIFv3 Confidence | ECS Conversion |
|---|---|
| Beyond Range | None |
| 0 - <3 | Low |
| 3 - <7 | Medium |
| 7 - 10 | High |
Exported fields
| Field | Description | Type |
|---|---|---|
| @timestamp | Event timestamp. | date |
| cif3.application | The application used by the indicator, such as telnet or ssh. | keyword |
| cif3.asn | AS Number of IP. | integer |
| cif3.asn_desc | AS Number org name. | keyword |
| cif3.cc | Country code of GeoIP. | keyword |
| cif3.city | GeoIP city information. | keyword |
| cif3.confidence | The confidence on a scale of 0-10 that the tags appropriately contextualize the indicator. | float |
| cif3.count | The number of times the same indicator has been reported with the same metadata by the same provider. | integer |
| cif3.deleted_at | The indicator expiration timestamp. | date |
| cif3.description | A description of the indicator. | keyword |
| cif3.expiration_duration | The configured expiration duration. | keyword |
| cif3.indicator | The value of the indicator, for example if the type is fqdn, this would be the value. | keyword |
| cif3.indicator_iprange | IPv4 or IPv6 IP Range. | ip_range |
| cif3.indicator_ipv4 | IPv4 address. | ip |
| cif3.indicator_ipv4_mask | subnet mask of IPv4 CIDR. | integer |
| cif3.indicator_ipv6 | singleton IPv6 address. | keyword |
| cif3.indicator_ipv6_mask | subnet mask of IPv6 CIDR. | integer |
| cif3.indicator_ssdeep_chunk | SSDEEP hash chunk. | text |
| cif3.indicator_ssdeep_chunksize | SSDEEP hash chunk size. | integer |
| cif3.indicator_ssdeep_double_chunk | SSDEEP hash double chunk. | text |
| cif3.itype | The indicator type, can for example be "ipv4, fqdn, email, url, sha256". | keyword |
| cif3.latitude | Latitude of GeoIP. | keyword |
| cif3.location | Lat/Long of GeoIP. | geo_point |
| cif3.longitude | Longitude of GeoIP. | keyword |
| cif3.portlist | The port or range of ports used by the indicator. | text |
| cif3.protocol | The protocol used by the indicator. | text |
| cif3.provider | The source of the indicator information. | keyword |
| cif3.rdata | Extra text or descriptive content related to the indicator such as OS, reverse lookup, etc. | keyword |
| cif3.reference | A reference URL with further info related to the indicator. | keyword |
| cif3.region | GeoIP region information. | keyword |
| cif3.tags | Comma-separated list of words describing the indicator such as "malware,exploit". | keyword |
| cif3.timezone | Timezone of GeoIP. | text |
| cif3.uuid | The ID of the indicator. | keyword |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
| data_stream.type | Data stream type. | constant_keyword |
| event.dataset | Event dataset | constant_keyword |
| event.module | Name of the module this data is coming from. | constant_keyword |
| input.type | Type of Filebeat input. | keyword |
| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword |
| log.flags | Flags for the log file. | keyword |
| log.offset | Offset of the entry in the log file. | long |
| threat.feed.name | Display friendly feed name | constant_keyword |
| threat.indicator.first_seen | The date and time when intelligence source first reported sighting this indicator. | date |
| threat.indicator.last_seen | The date and time when intelligence source last reported sighting this indicator. | date |
| threat.indicator.modified_at | The date and time when intelligence source last modified information for this indicator. | date |
| threat.indicator.tls.client.ja3 | An md5 hash that identifies clients based on their TLS handshake. | keyword |
Example
{
"@timestamp": "2024-08-01T08:05:14.040Z",
"agent": {
"ephemeral_id": "b351d699-2fd0-49f7-99e1-a7a471a29a62",
"id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.13.0"
},
"cif3": {
"deleted_at": "2022-09-03T20:25:53.000Z",
"expiration_duration": "45d",
"indicator": "20.206.75.106",
"itype": "ipv4",
"portlist": "443",
"uuid": "ac240898-1443-4d7e-a98a-1daed220c162"
},
"data_stream": {
"dataset": "ti_cif3.feed",
"namespace": "26457",
"type": "logs"
},
"ecs": {
"version": "8.11.0"
},
"elastic_agent": {
"id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
"snapshot": false,
"version": "8.13.0"
},
"event": {
"agent_id_status": "verified",
"category": [
"threat"
],
"created": "2024-08-01T08:05:14.040Z",
"dataset": "ti_cif3.feed",
"ingested": "2024-08-01T08:05:26Z",
"kind": "enrichment",
"original": "{\"application\":\"https\",\"asn\":8075,\"asn_desc\":\"microsoft-corp-msn-as-block\",\"cc\":\"br\",\"city\":\"campinas\",\"confidence\":10,\"count\":1,\"firsttime\":\"2022-07-20T20:25:53.000000Z\",\"group\":[\"everyone\"],\"indicator\":\"20.206.75.106\",\"indicator_ipv4\":\"20.206.75.106\",\"itype\":\"ipv4\",\"lasttime\":\"2022-07-20T20:25:53.000000Z\",\"latitude\":-22.9035,\"location\":[-47.0565,-22.9035],\"longitude\":-47.0565,\"portlist\":\"443\",\"protocol\":\"tcp\",\"provider\":\"sslbl.abuse.ch\",\"reference\":\"https://sslbl.abuse.ch/blacklist/sslipblacklist.csv\",\"region\":\"sao paulo\",\"reporttime\":\"2022-07-21T20:33:26.585967Z\",\"tags\":[\"botnet\"],\"timezone\":\"america/sao_paulo\",\"tlp\":\"white\",\"uuid\":\"ac240898-1443-4d7e-a98a-1daed220c162\"}",
"type": [
"indicator"
]
},
"input": {
"type": "httpjson"
},
"network": {
"protocol": "https",
"transport": "tcp"
},
"related": {
"ip": [
"20.206.75.106"
]
},
"tags": [
"preserve_original_event",
"forwarded",
"cif3-indicator",
"botnet"
],
"threat": {
"indicator": {
"as": {
"number": 8075,
"organization": {
"name": "microsoft-corp-msn-as-block"
}
},
"confidence": "High",
"first_seen": "2022-07-20T20:25:53.000Z",
"geo": {
"country_iso_code": "br",
"location": {
"lat": -22.9035,
"lon": -47.0565
},
"region_name": "sao paulo",
"timezone": "america/sao_paulo"
},
"ip": "20.206.75.106",
"last_seen": "2022-07-20T20:25:53.000Z",
"marking": {
"tlp": "WHITE"
},
"modified_at": "2022-07-21T20:33:26.585967Z",
"name": "20.206.75.106",
"provider": "sslbl.abuse.ch",
"reference": "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv",
"sightings": 1,
"type": "ipv4-addr"
}
}
}
Changelog
| Version | Details | Minimum Kibana version |
|---|---|---|
| 1.25.0 | Enhancement (View pull request) Update Elastic Managed integration documentation to the current wording and docs links. |
9.1.0 8.19.0 |
| 1.24.1 | Bug fix (View pull request) Statically map event.ingested in transform destination indices so detection rules using it as the timestamp override do not report empty destination indices as missing the field. |
9.1.0 8.19.0 |
| 1.24.0 | Enhancement (View pull request) Document the retention applied to source data streams on stateful and Serverless deployments, and how to change it. |
9.1.0 8.19.0 |
| 1.23.0 | Enhancement (View pull request) Remove redundant ECS field definitions and the catch-all dynamic template from the transforms, relying on the ecs@mappings component template. |
9.1.0 8.19.0 |
| 1.22.0 | Enhancement (View pull request) Enable request trace log removal. |
9.0.5 8.19.2 |
| 1.21.0 | Enhancement (View pull request) Add tags to ingest pipeline processors and add preserve_original_event to pipeline-level on_failure handlers. |
9.0.5 8.19.2 |
| 1.20.2 | Enhancement (View pull request) Set agentless deployment mode release field to ga. |
9.0.5 8.19.2 |
| 1.20.1 | Bug fix (View pull request) Fix transform destination to use keyword for data_stream.namespace, allowing multi-namespace deployments. |
9.0.5 8.19.2 |
| 1.20.0 | Enhancement (View pull request) Enable Agentless deployment. |
9.0.5 8.19.2 |
| 1.19.0 | Enhancement (View pull request) Use num_failure_retries instead of unattended mode for transform failure recovery. |
9.0.0 8.13.0 |
| 1.18.2 | Bug fix (View pull request) Use triple-brace Mustache templating when referencing variables in ingest pipelines. |
9.0.0 8.13.0 |
| 1.18.1 | Bug fix (View pull request) Remove duplicate security-solution-default tag references |
9.0.0 8.13.0 |
| 1.18.0 | Enhancement (View pull request) Allow transforms to run in unattended mode. |
9.0.0 8.13.0 |
| 1.17.0 | Enhancement (View pull request) Update Kibana constraint to support 9.0.0. |
9.0.0 8.13.0 |
| 1.16.0 | Enhancement (View pull request) Do not remove event.original in main ingest pipeline. |
8.13.0 |
| 1.15.0 | Enhancement (View pull request) Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error". |
8.13.0 |
| 1.14.4 | Bug fix (View pull request) Use triple-brace Mustache templating when referencing variables in ingest pipelines. |
8.13.0 |
| 1.14.3 | Bug fix (View pull request) Fix labels.is_ioc_transform_source values |
8.13.0 |
| 1.14.2 | Bug fix (View pull request) Add missing fields in transform |
8.13.0 |
| 1.14.1 | Bug fix (View pull request) Fix ECS date mapping on threat fields. |
8.13.0 |
| 1.14.0 | Enhancement (View pull request) Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. |
8.13.0 |
| 1.13.1 | Bug fix (View pull request) Adjust field mappings for transform destination index. |
8.12.0 |
| 1.13.0 | Enhancement (View pull request) Improve handling of empty responses. |
8.12.0 |
| 1.12.0 | Enhancement (View pull request) Support for IOC expiration |
8.12.0 |
| 1.11.0 | Enhancement (View pull request) Set sensitive values as secret. |
8.12.0 |
| 1.10.2 | Bug fix (View pull request) Clean up null handling |
8.7.1 |
| 1.10.1 | Enhancement (View pull request) Changed owners |
8.7.1 |
| 1.10.0 | Enhancement (View pull request) Limit request tracer log count to five. |
8.7.1 |
| 1.9.0 | Enhancement (View pull request) ECS version updated to 8.11.0. |
8.7.1 |
| 1.8.0 | Enhancement (View pull request) Set 'community' owner type. |
8.7.1 |
| 1.7.0 | Enhancement (View pull request) ECS version updated to 8.10.0. |
8.7.1 |
| 1.6.0 | Enhancement (View pull request) The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added 'owner.type: elastic' to package manifest. |
8.7.1 |
| 1.5.0 | Enhancement (View pull request) Add tags.yml file so that integration's dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI. |
8.7.1 |
| 1.4.0 | Enhancement (View pull request) Update package-spec to 2.9.0. |
8.7.1 |
| 1.3.0 | Enhancement (View pull request) Update package to ECS 8.9.0. |
8.7.1 |
| 1.2.0 | Enhancement (View pull request) Document duration units. |
8.7.1 |
| 1.1.0 | Enhancement (View pull request) Document valid duration units. |
8.7.1 |
| 1.0.0 | Enhancement (View pull request) Release Collective Intelligence Framework as GA. |
8.7.1 |
| 0.8.0 | Enhancement (View pull request) Ensure event.kind is correctly set for pipeline errors. |
8.7.1 |
| 0.7.0 | Enhancement (View pull request) Update package to ECS 8.8.0. |
8.7.1 |
| 0.6.0 | Enhancement (View pull request) Add a new flag to enable request tracing |
8.7.1 |
| 0.5.0 | Enhancement (View pull request) Update package to ECS 8.7.0. |
8.0.0 |
| 0.4.1 | Enhancement (View pull request) Honor preserve_original_event setting. |
8.0.0 |
| 0.4.0 | Enhancement (View pull request) Update package to ECS 8.6.0. |
8.0.0 |
| 0.3.1 | Bug fix (View pull request) Use ECS definition for threat.indicator.geo.location. |
8.0.0 |
| 0.3.0 | Enhancement (View pull request) Update package to ECS 8.5.0. |
— |
| 0.2.2 | Bug fix (View pull request) Remove duplicate field. |
8.0.0 |
| 0.2.1 | Enhancement (View pull request) Fix documentation build error |
8.0.0 |
| 0.2.0 | Enhancement (View pull request) Labelling with Threat Intelligence category |
8.0.0 |
| 0.1.0 | Enhancement (View pull request) Initial draft of the package |
8.0.0 |