Google Cloud Platform billing metricset

billing metricset is designed for collecting billing metrics from Google Cloud BigQuery daily cost detail table. BigQuery is a fully-managed, serverless data warehouse. Cloud Billing export to BigQuery enables you to export standard and detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify. Then you can access your Cloud Billing data from BigQuery for detailed analysis using Metricbeat. Please see export cloud billing data to BigQuery for more details on how to export billing data.

In BigQuery, Google Cloud daily cost data is categorized into two formats: standard and detailed. Each format is stored within a designated dataset and follows a structured schema for precise cost analysis. For a comprehensive understanding of these formats, consult the standard and detailed data schema documentation.

For standard usage cost data, set the table pattern format to gcp_billing_export_v1. This table pattern is set as the default when no other is specified.

For detailed usage cost data, set the table pattern to gcp_billing_export_resource_v1. Detailed tables include the standard fields and additional fields, such as effective_price, enabling a more granular view of expenses.

• dataset_id: (Required) Dataset ID that points to the top-level container which contains the actual billing tables.
• table_pattern: (Optional) Daily cost detail billing table name prefix. Default to gcp_billing_export_v1.
• cost_type: (Optional) The type of cost this line item represents: regular, tax, adjustment, or rounding error. Default to regular.

- module: gcp
  metricsets:
    - billing
  period: 24h
  project_id: "your project id"
  credentials_file_path: "your JSON credentials file path"
  dataset_id: "dataset id"
  table_pattern: "table pattern"
  cost_type: "regular"

For a description of each field in the metricset, see the exported fields section.

Here is an example document generated by this metricset:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud.account.id": "01475F-5B1080-1137E7",
    "cloud.project.id": "elastic-bi",
    "cloud.project.name": "elastic-containerlib-prod",
    "cloud.provider": "gcp",
    "event": {
        "dataset": "gcp.billing",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "billing": {
            "billing_account_id": "01475F-5B1080-1137E7",
            "cost_type": "regular",
            "invoice_month": "202106",
            "project_id": "containerlib-prod-12763",
            "project_name": "elastic-containerlib-prod",
            "total": 4717.170681,
            "sku_id": "0D56-2F80-52A5",
            "service_id": "6F81-5844-456A",
            "sku_description": "Network Inter Region Ingress from Jakarta to Americas",
            "service_description": "Compute Engine",
            "effective_price": 0.00292353,
            "tags": [
                {
                    "key": "size",
                    "value": "standard"
                }
            ]
        }
    },
    "metricset": {
        "name": "billing",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}