Skip to content

New bufferer component #5330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 13 commits into from
Closed

New bufferer component #5330

wants to merge 13 commits into from

Conversation

mapno
Copy link
Contributor

@mapno mapno commented Jun 27, 2025 •
edited
Loading

What this PR does:

The bufferer is a Kafka-based ingestion service that consumes trace data from Kafka topics instead of receiving it via gRPC like the traditional ingester. It reads serialized traces from Kafka, processes them through the same data pipeline as the ingester (live traces → head blocks → WAL → complete blocks).

                      ┌─────────────────────────────┐
                      │        Bufferer             │
                      │                             │
                      │                             │
                      │ • Manages instances map     │
                      │ • Shared WAL coordination   │
                      │ • Flush queue management    │
                      └─────────────┬───────────────┘
                                    │
                      ┌─────────────┴───────────────┐
                      │                             │
                      ▼                             ▼
          ┌─────────────────────┐       ┌─────────────────────┐
          │  PartitionReader    │       │   instances map     │
          │                     │       │   [tenantID]        │
          │ • Kafka consumer    │       │                     │
          │ • Watermark commits │       │ ┌─────────────────┐ │
          │ • Routes to consume │       │ |   instance A    │ │
          │   function          │       │ │   (tenant A)    │ │
          └─────────────────────┘       │ │                 │ │
                                        │ │ • Live traces   │ │
                                        │ │ • Head block    │ │
                                        │ │ • WAL blocks    │ │
                                        │ │ • Complete      │ │
                                        │ │   blocks        │ │
                                        │ └─────────────────┘ │
                                        │                     │
                                        │ ┌─────────────────┐ │
                                        │ │   instance B    │ │
                                        │ │   (tenant B)    │ │
                                        │ │                 │ │
                                        │ │ • Live traces   │ │
                                        │ │ • Head block    │ │
                                        │ │ • WAL blocks    │ │
                                        │ │ • Complete      │ │
                                        │ │   blocks        │ │
                                        │ └─────────────────┘ │
                                        └─────────────────────┘
                                                   │
                                                   ▼
                                        ┌─────────────────────┐
                                        │     Shared WAL      │
                                        │                     │
                                        │ • Single WAL for    │
                                        │   all instances     │
                                        │ • Coordinated       │
                                        │   block cutting     │
                                        └─────────────────────┘

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@mapno mapno force-pushed the rhythm-bufferer branch from 3511f27 to 5ec0217 Compare July 7, 2025 16:10
@mapno mapno force-pushed the rhythm-bufferer branch from 8f2c0bb to dc5a73e Compare July 8, 2025 13:26
@mapno mapno force-pushed the rhythm-bufferer branch from dc5a73e to 33fb555 Compare July 8, 2025 13:41
@mattdurham
Copy link
Contributor

In general does it make sense to move responsibility of timing and actions from the Bufferer to the individual instances? This would make the instances more self contained and potentially avoid a good chunk of the locking. Would be harder to do global cuts but we could do tenant specific then? Or trigger via a channel with at timer in the Bufferer.

@mattdurham
Copy link
Contributor

From chatting would calling Bufferer ReadCache make more sense?

@mattdurham
Copy link
Contributor

Can we document the failure modes and what happens? Do we need 100% data fidelity or can 99.9%, since the data will be correctly handled by long term storage regardless.

@mattdurham
Copy link
Contributor

Assuming failures happen is it better to return incorrect data or no data?

@mapno mapno mentioned this pull request Jul 21, 2025
Merged
3 tasks
@mapno
Copy link
Contributor Author

mapno commented Jul 21, 2025

Development has been moved to a dev branch. See #5430

@mapno mapno closed this Jul 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@knylander-grafana knylander-grafana Awaiting requested review from knylander-grafana knylander-grafana will be requested when the pull request is marked ready for review knylander-grafana is a code owner

@joe-elliott joe-elliott Awaiting requested review from joe-elliott joe-elliott will be requested when the pull request is marked ready for review joe-elliott is a code owner

@mdisibio mdisibio Awaiting requested review from mdisibio mdisibio will be requested when the pull request is marked ready for review mdisibio is a code owner

@yvrhdn yvrhdn Awaiting requested review from yvrhdn yvrhdn will be requested when the pull request is marked ready for review yvrhdn is a code owner

@zalegrala zalegrala Awaiting requested review from zalegrala zalegrala will be requested when the pull request is marked ready for review zalegrala is a code owner

@electron0zero electron0zero Awaiting requested review from electron0zero electron0zero will be requested when the pull request is marked ready for review electron0zero is a code owner

@ie-pham ie-pham Awaiting requested review from ie-pham ie-pham will be requested when the pull request is marked ready for review ie-pham is a code owner

@stoewer stoewer Awaiting requested review from stoewer stoewer will be requested when the pull request is marked ready for review stoewer is a code owner

@javiermolinar javiermolinar Awaiting requested review from javiermolinar javiermolinar will be requested when the pull request is marked ready for review javiermolinar is a code owner

@carles-grafana carles-grafana Awaiting requested review from carles-grafana carles-grafana will be requested when the pull request is marked ready for review carles-grafana is a code owner

@ruslan-mikhailov ruslan-mikhailov Awaiting requested review from ruslan-mikhailov ruslan-mikhailov will be requested when the pull request is marked ready for review ruslan-mikhailov is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mapno @mattdurham