[text index] don't optimize when loading a mmap-backed immutable index #6785
+25
−67
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
📝 WalkthroughWalkthroughThe changes update the conversion process from Possibly related PRs
Suggested reviewers
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🧰 Additional context used🧠 Learnings (2)📓 Common learningslib/segment/src/index/field_index/full_text_index/immutable_text_index.rs (3)⏰ Context from checks skipped due to timeout of 90000ms (14)
🔇 Additional comments (1)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
lib/segment/src/index/field_index/full_text_index/inverted_index/immutable_inverted_index.rs (1)
397-410: Consider the implications of including empty postingsThe simplified conversion no longer filters out empty postings, which differs from the
From<MutableInvertedIndex>implementation that optimizes by removing them. This inconsistency might lead to:
- Increased memory usage for empty postings
- Different behavior between mutable-to-immutable vs mmap-to-immutable conversions
While this aligns with the PR objective of avoiding optimization during mmap loading, it creates an inconsistency in how immutable indexes are constructed from different sources.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
lib/segment/src/index/field_index/full_text_index/inverted_index/immutable_inverted_index.rs(1 hunks)lib/segment/src/index/field_index/full_text_index/inverted_index/mmap_inverted_index/mmap_postings.rs(1 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: generall
PR: qdrant/qdrant#6766
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mutable_inverted_index.rs:274-282
Timestamp: 2025-06-28T19:25:53.421Z
Learning: In the full text inverted index implementations (MutableInvertedIndex, ImmutableInvertedIndex, MmapInvertedIndex), there's an important distinction between TokenId and PointOffsetType: TokenId represents a token/word identifier used to index into postings lists, while PointOffsetType represents a point/document identifier. The iter_point_ids method should take a TokenId and return an iterator over PointOffsetType values (points that contain that token), not the other way around.
Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.
Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/iterator.rs:0-0
Timestamp: 2025-05-26T14:50:13.183Z
Learning: The `advance_until_greater_or_equal` method in `PostingIterator` in `lib/posting_list/src/iterator.rs` is intentionally designed with different semantics than `next()` and can output the same value multiple times. This is intended behavior for a search method that finds the first element with ID >= target_id.
lib/segment/src/index/field_index/full_text_index/inverted_index/mmap_inverted_index/mmap_postings.rs (4)
Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.
Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/iterator.rs:0-0
Timestamp: 2025-05-26T14:50:13.183Z
Learning: The `advance_until_greater_or_equal` method in `PostingIterator` in `lib/posting_list/src/iterator.rs` is intentionally designed with different semantics than `next()` and can output the same value multiple times. This is intended behavior for a search method that finds the first element with ID >= target_id.
Learnt from: generall
PR: qdrant/qdrant#6766
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mutable_inverted_index.rs:274-282
Timestamp: 2025-06-28T19:25:53.421Z
Learning: In the full text inverted index implementations (MutableInvertedIndex, ImmutableInvertedIndex, MmapInvertedIndex), there's an important distinction between TokenId and PointOffsetType: TokenId represents a token/word identifier used to index into postings lists, while PointOffsetType represents a point/document identifier. The iter_point_ids method should take a TokenId and return an iterator over PointOffsetType values (points that contain that token), not the other way around.
Learnt from: generall
PR: qdrant/qdrant#6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The `populate()` method in `QuantizedMmapStorage` correctly returns void instead of a `Result` because it directly implements the `Madviseable` trait which defines `populate(&self)` without a return type. Adding an unnecessary `Ok(())` return would trigger Clippy warnings about unnecessary Result wrapping.
lib/segment/src/index/field_index/full_text_index/inverted_index/immutable_inverted_index.rs (4)
Learnt from: generall
PR: qdrant/qdrant#6766
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mutable_inverted_index.rs:274-282
Timestamp: 2025-06-28T19:25:53.421Z
Learning: In the full text inverted index implementations (MutableInvertedIndex, ImmutableInvertedIndex, MmapInvertedIndex), there's an important distinction between TokenId and PointOffsetType: TokenId represents a token/word identifier used to index into postings lists, while PointOffsetType represents a point/document identifier. The iter_point_ids method should take a TokenId and return an iterator over PointOffsetType values (points that contain that token), not the other way around.
Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.
Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/iterator.rs:0-0
Timestamp: 2025-05-26T14:50:13.183Z
Learning: The `advance_until_greater_or_equal` method in `PostingIterator` in `lib/posting_list/src/iterator.rs` is intentionally designed with different semantics than `next()` and can output the same value multiple times. This is intended behavior for a search method that finds the first element with ID >= target_id.
Learnt from: coszio
PR: qdrant/qdrant#6609
File: lib/gridstore/src/blob.rs:46-59
Timestamp: 2025-06-02T18:10:47.203Z
Learning: In the Qdrant codebase, zerocopy crate is extensively used for safe byte-level operations across GPU operations, HNSW indices, memory-mapped structures, and serialization. When implementing Blob trait for generic Vec<T>, using zerocopy's FromBytes and IntoBytes traits is preferred over size_of::<T>() because it guarantees memory layout equals byte representation, making serialization safe and correct.
⏰ Context from checks skipped due to timeout of 90000ms (14)
- GitHub Check: test-snapshot-operations-s3-minio
- GitHub Check: test-shard-snapshot-api-s3-minio
- GitHub Check: test-consistency
- GitHub Check: test-consensus-compose
- GitHub Check: test-low-resources
- GitHub Check: Basic TLS/HTTPS tests
- GitHub Check: integration-tests-consensus
- GitHub Check: integration-tests
- GitHub Check: rust-tests (macos-latest)
- GitHub Check: rust-tests (ubuntu-latest)
- GitHub Check: storage-compat-test
- GitHub Check: rust-tests (windows-latest)
- GitHub Check: rust-tests-no-rocksdb (ubuntu-latest)
- GitHub Check: lint
🔇 Additional comments (2)
lib/segment/src/index/field_index/full_text_index/inverted_index/mmap_inverted_index/mmap_postings.rs (1)
312-315: Assumption of existing posting lists is valid
Theposting_countinPostingsHeaderexactly matches the number of serializedPostingListHeaderentries, andMmapPostings::getwill only returnNonefor out-of-range token IDs or a corrupted/truncated file (which should never occur for a correctly written index). The currentfilter_map—and its accompanying comment—can remain as-is.lib/segment/src/index/field_index/full_text_index/inverted_index/immutable_inverted_index.rs (1)
418-421: Debug assertion is safe—no length mismatch possible
Thefilter_mapinMmapPostings::iter_postingsonly skips out-of-bounds entries (never actual token IDs), so it yields exactly one view per stored posting. And inMmapInvertedIndex::createwe alreadydebug_assert_eq!(vocab.len(), postings.len())when writing the index. As a result, the in-memory conversion inimmutable_inverted_index.rs(lines 418–421) will always seepostings.len() == vocab.len(). No changes required.Likely an incorrect or invalid review comment.
Comment on lines
+412
to
+416
| let vocab: HashMap<String, TokenId> = index | ||
| .vocab | ||
| .iter() | ||
| .map(|(token_str, token_id)| (token_str.to_owned(), token_id[0])) | ||
| .collect(); |
There was a problem hiding this comment.
Potential issue with vocabulary token ID mapping
The code assumes token_id[0] always exists when building the vocabulary. If the token ID array is empty, this will panic. Consider adding a check or using token_id.first() with proper error handling.
Apply this diff to handle empty token ID arrays safely:
- let vocab: HashMap<String, TokenId> = index
- .vocab
- .iter()
- .map(|(token_str, token_id)| (token_str.to_owned(), token_id[0]))
- .collect();
+ let vocab: HashMap<String, TokenId> = index
+ .vocab
+ .iter()
+ .filter_map(|(token_str, token_id)| {
+ token_id.first().map(|&id| (token_str.to_owned(), id))
+ })
+ .collect();📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| let vocab: HashMap<String, TokenId> = index | |
| .vocab | |
| .iter() | |
| .map(|(token_str, token_id)| (token_str.to_owned(), token_id[0])) | |
| .collect(); | |
| let vocab: HashMap<String, TokenId> = index | |
| .vocab | |
| .iter() | |
| .filter_map(|(token_str, token_id)| { | |
| token_id.first().map(|&id| (token_str.to_owned(), id)) | |
| }) | |
| .collect(); |
🤖 Prompt for AI Agents
In
lib/segment/src/index/field_index/full_text_index/inverted_index/immutable_inverted_index.rs
around lines 412 to 416, the code assumes token_id[0] always exists, which can
cause a panic if the token ID array is empty. To fix this, replace token_id[0]
with token_id.first() and add proper error handling or filtering to safely
handle empty arrays, ensuring no panics occur during vocabulary construction.
generall
approved these changes
Jul 1, 2025
|
After loading from mmap into ram we should clear mmap cache |
coszio
commented
Jul 1, 2025
Comment on lines
128
to
132
| index.inverted_index.clear_cache()?; | ||
|
|
||
| // Index is now loaded into memory, clear cache of backing mmap storage | ||
| if let Err(err) = index.clear_cache() { | ||
| log::warn!("Failed to clear mmap cache of ram mmap full text index: {err}"); |
There was a problem hiding this comment.
This was already done the line below 😄
This reverts commit 110199e.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In
dev, loading a mmap-backed immutable index involved optimizing in the same way as when mutable turns immutable. This removed posting lists which have no items, and tries to reduce vocabulary size.However, in the case of mmap->immutable, I don't think this is critical, and requires practically rebuilding the index on load.
This PR, makes it possible to just turn the mmap structures into in-memory immutable ones by copying the content.