use sequential reads for building quantized vectors #6796

generall · 2025-07-02T21:38:27Z

No description provided.

coderabbitai · 2025-07-02T21:40:47Z

📝 Walkthrough

"""

Walkthrough

A new method, get_dense_sequential, was added to the DenseVectorStorage trait and implemented across its primary storage backends, including appendable mmap, memmap, simple, and volatile dense vector storages, as well as in test fixtures. This method provides sequential access to dense vector slices for a given key and is functionally similar to the existing get_dense method. The method is also used to replace previous calls to get_dense in multi-dense vector storage iteration and quantized vector creation. Additionally, a new populate_vector_storages method was added to the SegmentBuilder to preload vector caches, and this method is invoked in the segment optimizer before CPU-bound indexing begins.

Possibly related PRs

Add volatile dense vector storage, use in tests #6561: The main PR adds a new method get_dense_sequential to various DenseVectorStorage implementations, including VolatileDenseVectorStorage, which is introduced and extensively tested in the retrieved PR that adds the volatile dense vector storage type for testing purposes.
Batched reader for segment construction #6487: The main PR adds a get_dense_sequential method to the DenseVectorStorage implementation for TestRawScorerProducer, while the retrieved PR adds a similarly named but distinct get_vector_sequential method to the VectorStorage implementation for the same struct; although both methods relate to sequential vector access in the same file and struct, they belong to different traits (DenseVectorStorage vs VectorStorage) and serve different return types, indicating related but not identical functionality.

Suggested reviewers

generall
"""

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Clippy (1.86.0)

Updating git repository `https://github.com/qdrant/tar-rs`

error: failed to load source for dependency tar

Caused by:
Unable to update https://github.com/qdrant/tar-rs?branch=main#856dbd09

Caused by:
failed to create directory /usr/local/git/db/tar-rs-f03560d008573c0e

Caused by:
Permission denied (os error 13)

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 645966e and 4f0713e.

📒 Files selected for processing (1)

lib/segment/src/fixtures/index_fixtures.rs (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

lib/segment/src/fixtures/index_fixtures.rs

⏰ Context from checks skipped due to timeout of 90000ms (14)

GitHub Check: test-consistency
GitHub Check: test-low-resources
GitHub Check: test-consensus-compose
GitHub Check: test-snapshot-operations-s3-minio
GitHub Check: Basic TLS/HTTPS tests
GitHub Check: test-shard-snapshot-api-s3-minio
GitHub Check: integration-tests-consensus
GitHub Check: lint
GitHub Check: integration-tests
GitHub Check: rust-tests (ubuntu-latest)
GitHub Check: rust-tests-no-rocksdb (ubuntu-latest)
GitHub Check: storage-compat-test
GitHub Check: rust-tests (macos-latest)
GitHub Check: rust-tests (windows-latest)

✨ Finishing Touches

📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

xzfc · 2025-07-02T22:00:50Z

lib/segment/src/segment_constructor/segment_builder.rs

@@ -732,6 +732,14 @@ impl SegmentBuilder {
        }
        Ok(quantized_vectors_map)
    }
+
+    /// Populate cache of all vector storages, so it will be faster to build index
+    pub fn cache_vectors(&self) -> OperationResult<()> {


Nit: consistent naming

Suggested change

pub fn cache_vectors(&self) -> OperationResult<()> {

pub fn populate_vector_storages(&self) -> OperationResult<()> {

generall · 2025-07-03T07:37:59Z

Now it is saturating IO:

lib/segment/src/fixtures/index_fixtures.rs

* use sequential reads for building quantized vectors * populate vector storages before indexing * consistent naming * Update lib/segment/src/fixtures/index_fixtures.rs --------- Co-authored-by: Tim Visée <tim+github@visee.me>

use sequential reads for building quantized vectors

c882005

generall requested a review from xzfc July 2, 2025 21:38

xzfc approved these changes Jul 2, 2025

View reviewed changes

populate vector storages before indexing

295ad62

xzfc reviewed Jul 2, 2025

View reviewed changes

consistent naming

645966e

timvisee reviewed Jul 3, 2025

View reviewed changes

lib/segment/src/fixtures/index_fixtures.rs Show resolved Hide resolved

timvisee approved these changes Jul 3, 2025

View reviewed changes

Update lib/segment/src/fixtures/index_fixtures.rs

4f0713e

generall merged commit 485eeed into dev Jul 3, 2025
18 checks passed

generall deleted the fix-reading-vectors-in-quantization branch July 3, 2025 10:15

This was referenced Aug 30, 2025

Generic AccessPattern #7166

Merged

Create and load for an appendable quantization #7193

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

use sequential reads for building quantized vectors #6796

use sequential reads for building quantized vectors #6796

Uh oh!

generall commented Jul 2, 2025

Uh oh!

coderabbitai bot commented Jul 2, 2025 •

edited

Loading

Walkthrough

Possibly related PRs

Suggested reviewers

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

Uh oh!

xzfc Jul 2, 2025

Uh oh!

generall commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

	pub fn cache_vectors(&self) -> OperationResult<()> {
	pub fn populate_vector_storages(&self) -> OperationResult<()> {

use sequential reads for building quantized vectors #6796

use sequential reads for building quantized vectors #6796

Uh oh!

Conversation

generall commented Jul 2, 2025

Uh oh!

coderabbitai bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Possibly related PRs

Suggested reviewers

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

xzfc Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

generall commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai bot commented Jul 2, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)