Proposal

This issue covers outstanding issues and explains the proposed longer term plan for Created Timestamp feature in Prometheus Ecosystem.

Background

In 2023 we had amazing @ArthurSens as our GSoC mentee to introduce Created Timestamp feature (CT). We delivered initial piping in Prometheus proto format, client_golang and initial Prometheus feature called created-timestamp-zero-ingestion. We also delivered a PromCon 2023 talk explaining Created Timestamp. Last, but no least, we gained super active and talented maintainer @ArthurSens

However, this is only the beginning for full productionizing and adoption of Created Timestamp. As mentioned in the CT proposal @ArthurSens did, the zero injection will be accompanied (or replaced) by metadata storage and there further various work items that are still missing.

In 2024 summer we have @Maniktherana on another round of GSoC mentorship around CT feature 🤗

Motivation

Reminder of why we do CTs:

• More accurate rates
• Support OTLP protocol one day.
• Support backends which require CT for accuracy e.g. Google Cloud Monitoring.

Issues

Following work is planned. A bit of rationales are provided for prioritization purposes:

• PRW spec change: remote write 2.0: Add support for created timestamp propagation [spec] #13368
• Fix final known limitations of OM text CT parsing (TestCTParseFailures)
• Optimize OM text CT parsing Created Timestamp: Optimize memory use for OM Text CT parsing (with created-timestamp-zero-ingestion feature flag) #14823
• Scrape loop test and skipping CT metrics when CT injection is set on https://github.com/prometheus/prometheus/pull/13506/files#diff-94ac40a66b06f7a86e45c1415b90cece99f310db590a8dd2b1310a5a81c21996R719
  • Solves Handle OpenMetrics ..._created lines #6541
• Parse CT from OM Text: Parse Created timestamps from OpenMetrics text format #12980
  • This also solves: Store metadata for OpenMetrics _created metrics #10164
• CT zero ingestion for native histograms: tsdb: need histogram support for created-timestamp handling #13384
• Propose alternative CT attachment to OM Text: Propose alternative Created Timestamp handling for OpenMetrics Text Format #13537
  • This solves known drawbacks of _created series mechanism.
• Store CT in WAL storage: Created Timestamp: Store CT in WAL #14218
  • The this solves known drawbacks with created-timestamp-zero-ingestion feature. Potentially allow deprecating or at least reduce the use of zero ingestion feature.
  • Use CT from metadata storage in Remote Write 2.0: Created Timestamp: Prometheus progates CT in Remote-Write 2.0 #14220
  • Use CT from metadata storage in PromQL engine: Created Timestamp: Use CT from metadata storage in PromQL engine #14219
  • Related docs https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/wal.md#metadata-records
• Add optional feature for auto-creation of CT: Created Timestamp: Opt-in CT auto-generation globally/per scrape job. #14763
• Improvement of rates with process start use of CT in Otel Cumulative data points should set StartTimeUnixNano per timeseries open-telemetry/opentelemetry-specification#4184
• OTLP ingestion use of CT Handle OTLP StartTime (similar to OpenMetrics Created timestamp) #14600