Improve shard transfer tracker, show fraction in case of resharding #6084
Conversation
| // - shards: 4 -> 3 | ||
| // points: 25/25/25/25 -> 33/33/33 | ||
| // transfer fraction of each shard: 1/3 = 0.333 | ||
| Ordering::Greater => 1.0 / to as f32, |
There was a problem hiding this comment.
How do we know that to can't be zero here?
There was a problem hiding this comment.
We can't have 0 shards. It is rejected on a higher level, and thus you cannot shard down to 0 shards either.
|
|
||
| pub fn add(&mut self, delta: usize) { | ||
| self.points_transferred += delta; | ||
| self.points_total = max(self.points_total, self.points_transferred); |
There was a problem hiding this comment.
Why is points_total changing during the resharding?
Is it to account for some ongoing updates?
There was a problem hiding this comment.
The total count is an estimate, as it's very expensive to do exact counting here.
It therefore is expected to overshoot the estimated total about 50% of the time.
Note that since #6074 we use non-exact counting in regular shard transfers as well in which case the same might happen.
I'd prefer to update the total instead of showing some broken value like 123/100.
…6084) * Improve transfer progress tracker, use add and set functions * Count actually transferred points in batch * While resharding, estimate transfer size based on shard fraction * Remove debug assertion causing test failures * Reformat
Extension of #6074
Improve tracking of records to transfer in shard transfers, specifically for stream records and resharding stream records transfer.
The main two improvements are:
For example, if we're resharding from one to two shards, a resharding transfer will only transfer ~50% of the records, not all of them.
The current resharding implementation still filters points with the hash ring after scrolling, not during scrolling itself. Using the hash ring filter during scrolling itself is very expensive and makes the transfer significantly slower, bumping transfer time from 28 to 500 seconds.
Branch with it implemented: https://github.com/qdrant/qdrant/commits/improve-transfer-progress-tracker-hashring-filter/
Example collection cluster info during resharding transfer (now shows 250k and correct record count):
{ "peer_id": 6294971774567105, "shard_count": 2, "local_shards": [ { "shard_id": 0, "points_count": 500000, "state": "Active" }, { "shard_id": 1, "points_count": 145399, "state": "Resharding" } ], "shard_transfers": [ { "shard_id": 0, "to_shard_id": 1, "from": 6294971774567105, "to": 6294971774567105, "sync": true, "method": "resharding_stream_records", "comment": "Transferring records (145399/250000), started 16s ago, ETA: 11.68s" } ], // ... }All Submissions:
devbranch. Did you create your branch fromdev?New Feature Submissions:
cargo +nightly fmt --allcommand prior to submission?cargo clippy --all --all-featurescommand?