This project integrates Confluent Cloud Cost information with third parties to help on data aggregation related to service usage or cost breakdowns. It can work as a Service exposing cost information as another Confluent Cloud metric so it can be integrated with Prometheus and Grafana or as a CRON job that pushes cost information to a given target.

• Confluent Cloud Metrics API provides metrics for Confluent Cloud resources, supports Prometheus export and filtering by resource.

• Confluent Cloud Costs API provides costs for Confluent Cloud resources.

Confluent Costs API considerations:

• Cost data can take up to 72 hours to become available
• Start date can reach a maximum of one year into the past
• One month is the maximum window between start and end dates.

Confluent Cloud cost exporter is a GIN server that exposes the Confluent Cloud Costs API as Prometheus metrics or JSON Metrics for Metricbeat. Confluent Cloud costs exporter uses an internal in memory cache to reduce the number of requests to Confluent Cloud Cost API, cache expiration configuration needs to be used to define the request frequency to Confluent Cost API, bigger cache expiration time means less external calls. For example, Prometheus works better with scrape_interval congigured with less than 5m, but the cost data do not change so frequently, the costs exporters use the cache to serve data to Prometheus or Metricbeat.

On the other side, the JSON exporter helps on data types normalization, avoiding the Try to convert long to float Elasticsearch error when tried to use Metricbeat HTTP module with Confluent Cloud Costs API endpoint.

A given period is required for building the query to the Cost API endpoint, the exporter by default will work getting data from three days ago for only one day.

period:
  daysAgo: 3
  window: 1

Additionally, Confluent Cloud cost exporter can be configured to act as a CRON job to push costs to a given target.

Supported CRON targets:

• Confluent Cloud Kafka Topic (AVRO schema).

./confluent_cost_exporter Readme.md

Needs Confluent Cloud credentials to retrieve costs. Service account must be a member of the OrganizationAdmin role.

More information

Environment variables:

• CONFLUENT_CLOUD_API_KEY - Confluent Cloud API Key
• CONFLUENT_CLOUD_API_SECRET - Confluent Cloud API Secret

Default exporter is the web exporter, it will expose cost information in the following endpoints:

• /probe: Confluent Cloud Cost as prometheus
• /json: Confluent Cloud Cost as json

Configure cache expiration. Default: 30m

• CACHE_EXPIRATION - Cache expiration time. Default: 30m

Prometheus interval is set to 5m and timeout to 30s. More than a 5m interval is not recommended for Prometheus.

prometheus.yml file is configured to scrape metrics from Confluent Cloud Metrics API and from Confluent Cloud Cost exporter.

 - job_name: Confluent Cloud
    scrape_interval: 1m
    scrape_timeout: 1m
    honor_timestamps: true
    static_configs:
      - targets:
        - api.telemetry.confluent.cloud
    scheme: https
    basic_auth:
      username: <CONFLUENT_API_KEY>
      password: <CONFLUENT_API_SECRET>
    metrics_path: /v2/metrics/cloud/export
    params:
      resource.kafka.id:
        - lkc-<id>
      resource.schema_registry.id:
        - lsrc-<id>

Getting Confluent cloud cost as Prometheus Metric, confluent_cloud_cost_amount with labels:

Example:

# HELP confluent_cloud_cost_amount Confluent Cloud Resource costs
# TYPE confluent_cloud_cost_amount untyped
confluent_cloud_cost_amount{discount="0",end_date="2023-10-02",environment="env-xxxxxx",granularity="DAILY",id="lkc-xxxxxx",original_amount="0",price="0.0001326",product="KAFKA",quantity="2.6151538e-05",resource="connect",start="2023-10-01",unit="GB-hour"} 0
confluent_cloud_cost_amount{discount="0",end_date="2023-10-02",environment="env-xxxxxx",granularity="DAILY",id="lkc-xxxxxx",original_amount="3.1368",price="0.00484",product="KAFKA",quantity="648",resource="connect",start="2023-10-01",unit="Partition-hour"} 3.1368

Prometheus Job:

  - job_name: confluent_cost_exporter
    scrape_interval: 5m
    scrape_timeout: 30s
    honor_labels: true 
    metrics_path: /probe  
    static_configs:
      - targets: ['confluent_cost_exporter:7979'] 

Validate Targets are up.

Open http://localhost:9090/targets?search=

Provisioned datasource: (./grafana/datasources/datasource.yml)

Open http://localhost:3000 and login as admin.

Provisioned dashboards: (./grafana/dashboards)

• Confluent Cloud.

• Confluent Cloud Cost.

MetricBeat and Elasticsearch: It is possible to use Metribeat HTTP module to get the data from Confluent Cloud Costs API, but it requires to manage type conversions defining an Elasticsearch index template.

Issue: Metricbeat HTTP module. Try to convert long to float.

• Metricbeat will try to insert into Elasticsearch quantity: 20 as long.

• Metricbeat will try to insert into Elasticsearch quantity: 10.4 as float.

Confluent Cloud Cost exporter will convert the quantity to float, forcing the quantity to be a float number always serialized with dot . as decimal separator.

mtericbeat.yml file is configured to collect metrics from Confluent Cloud Metrics API and send them to Elasticsearch.

Confluet Cloud Metrics:

metricbeat.modules:
 - module: prometheus
    period: 900s 
    hosts: ['https://api.telemetry.confluent.cloud/v2/metrics/cloud/export?resource.kafka.id=lkc-q8dr5m&resource.kafka.id=lkc-12771v&resource.schema_registry.id=lsrc-nvmj2d']
    fields_under_root: true
    username: <CONFLUENT_CLOUD_API_KEY>
    password: <CONFLUENT_CLOUD_API_SECRET>

Confluent Cloud Cost as Metricbeat metric.

metricbeat.modules:
  #------------------------- Confluent Cloud Cost Collector Module --------------------- 
  - module: http
    period: 5m 
    metricsets:
     - json
    hosts: ['http://confluent_cost_exporter:7979']
    namespace: "confluent"
    path: "/json" 
    method: "GET"
    json.is_array: true

Import Kibana dashboards:

• Confluent Cloud Metrics & Costs: confluent_cloud_metrics_costs.ndjson

TODO: Auto-provision Kinana dashboards.

Define required environment variables at docker-compose.yaml file.

Confluent Cloud Cost exporter:

environment:
    - CONFLUENT_CLOUD_API_KEY=<CONFLUENT_CLOUD_API_KEY>
    - CONFLUENT_CLOUD_API_SECRET=<CONFLUENT_CLOUD_API_SECRET>
    - CACHE_EXPIRATION=240m

1. Copy or rename docker-compose-pg.yaml to docker-compose.yml to start Prometheus, Grafana and the Confluent Cost exporter.

2. Edit docker-compose.yml and fill the required environment variables.

3. Create a env folder, compose file will mount env folder for managing services configuration:

4. Create env/prometheus.yml for Prometheus jobs configuration. (use prometheus/prometheus-template.yml as reference)

5. Build: docker-compose build

6. Up: docker-compose up -d

1. Copy docker-compose-elk.yaml to docker-compose.yml to start Elasticsearch, Kibana, Metricbeat and the Confluent Cost exporter.

2. Edit docker-compose.yml and fill the required environment variables.

3. Create a env folder, compose file will mount env folder for managing services configuration:

4. Create env/metricbeat.yml for Prometheus jobs configuration. (use metricbeat/metricbeat-template.yml as reference)

5. Build: docker-compose build

6. Up: docker-compose up -d

Enabling CRON exporters will run the exporters as CRON jobs, web exporters configuration will be ignored.

Environment variable:

• CRON_EXPRESSION=<CRON_EXPRESSION>

Environment variables:

• BOOTSTRAP=<CONFLUENT_CLOUD_BOOTSTRAP_SERVER>
• CONFLUENT_CLOUD_API_KEY=<CONFLUENT_CLOUD_API_KEY>
• CONFLUENT_CLOUD_API_SECRET=<CONFLUENT_CLOUD_API_SECRET>
• TOPIC=<TOPIC_NAME>
• SCHEMA_REGISTRY_ENDPOINT=<SCHEMA_REGISTRY_ENDPOINT>
• SCHEMA_REGISTRY_API_KEY=<SCHEMA_REGISTRY_API_KEY>
• SCHEMA_REGISTRY_API_SECRET=<SCHEMA_REGISTRY_API_SECRET>

Using a configuration file:

credentials: 
  key: <CONFLUENT_CLOUD_API_KEY>
  secret: <CONFLUENT_CLOUD_API_SECRET>
cron:
  expression: "<CRON_EXPRESSION>" 
  target:
    kafka:
      bootstrap: <BOOTSTRAP_SERVER>
      credentials: 
        key: <CONFLUENT_CLOUD_KAFKA_API_KEY>
        secret: <CONFLUENT_CLOUD_KAFKA_API_SECRET>
      schemaRegistry:
        endpoint: <SCHEMA_REGISTRY_ENDPOINT>
        credentials:
          key: <SCHEMA_REGISTRY_API_KEY>
          secret: <SCHEMA_REGISTRY_API_SECRET>
      topic: <TOPIC_NAME>

Using a configuration file:

docker run -p 7979:7979 -v <path/to/config.yml>:/bin/config.yml confluent/confluent_cloud_cost_exporter 

• Grafana. Add more Panels to the dashboards
• Grafana. Combine Metrics and Costs in the same dashboard
• Auto-provision Kinana dashboards.
• Alerting
• Exporters