Skip to content

Configurable shardedPostings sharding function #13544

@yeya24

Description

@yeya24

Proposal

#10421 added a new method ShardedPostings in IndexReader to filter a specific shard using provided shardIndex and shardCount parameter.

The implementation in #10421 uses hashmod to check if a series belong to the given shard. However, there are different ways to do that and hashmod is only one way. It would be good to use a configurable function to customize the sharding logic. This might not be used in Prometheus, but it would be useful for downstream projects.

There are two main implementations of ShardedPostings method:

  1. index.Reader https://github.com/prometheus/prometheus/blob/main/tsdb/index/index.go#L1774
  2. headIndexReader https://github.com/prometheus/prometheus/blob/main/tsdb/head_read.go

I am still unsure where I can add the logic to allow passing a custom function from downstream projects, especially for index.Reader. Maybe I have to change ShardedPostings method signature to

ShardedPostings(p index.Postings, shardIndex, shardCount uint64, shardFunc func(hash, shardIndex, shardCount uint64) bool) index.Postings

Any suggestion would be appreciated!

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions