MADV_POPULATE_READ on sequential mmaps #6923

xzfc · 2025-07-22T09:54:23Z

It turns out MADV_POPULATE_READ performance is influenced by the MADV_RANDOM and MADV_SEQUENTIAL flags.
Obvious in hindsight, but still surprising that the kernel won't enable readahead unconditionlly given that it knows that it will be reading the whole file.

// Slow, many 4K IOPS
madvise(ptr, size, MADV_RANDOM);
madvise(ptr, size, MADV_POPULATE_READ);

// Fast, as uses large readahead
madvise(ptr, size, MADV_SEQUENTIAL);
madvise(ptr, size, MADV_POPULATE_READ);

Profiler

It found using this profiler: https://github.com/xzfc/qdrant-tools/tree/3934acc179c9187a6b90d470de9a48fd0a9dd3c9/bpf-mfault-paths

Fix

Since a lot of our files mmaped twice (for sequential and for random), the fix in this PR is trivial: let populate() use the sequential mmap if it available.

Fixed:

Dense vector storages, except deleted flags
Gridstore pages (used in sparse vectors)

Some mmaps are not duplicated, so these are not fixed:

MmapBitSlice (deleted flags).
Gridstore bitmasks (bitmask.dat and gaps.dat; used in sparse vectors)
Quantized vector storage. (not used)
Payload storages. (not used)

CPU usage/saturation on 400M benchmark (left is before this commit, right is after):

coderabbitai · 2025-07-22T09:55:53Z

📝 Walkthrough

Walkthrough

This change modifies the internal implementation of the populate method across several memory-mapped data structures. Specifically, the method is updated to call populate() on read-only or sequentially advised mmap fields (such as mmap_seq, mmap_sequential, or mmap_seq), instead of the previous writable or default mmap fields. Additionally, new populate methods are introduced in the MmapTypeReadOnly<[T]> and MmapSliceReadOnly<T> structs to support this behavior. No public API signatures are changed; all modifications are internal to method implementations.

Estimated code review effort

2 (~15 minutes)

Possibly related PRs

disk cache hygiene #6323: Introduces populate and clear_cache methods across mmap-based components and removes delayed mmap population, directly relating to the changes in this PR regarding memory population strategies.

Suggested reviewers

timvisee

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ac852aa and 67262c4.

📒 Files selected for processing (4)

lib/common/memory/src/chunked_utils.rs (1 hunks)
lib/common/memory/src/mmap_type_readonly.rs (3 hunks)
lib/gridstore/src/page.rs (1 hunks)
lib/segment/src/vector_storage/dense/mmap_dense_vectors.rs (1 hunks)

🧠 Learnings (5)

📓 Common learnings

Learnt from: generall
PR: qdrant/qdrant#6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The `populate()` method in `QuantizedMmapStorage` correctly returns void instead of a `Result` because it directly implements the `Madviseable` trait which defines `populate(&self)` without a return type. Adding an unnecessary `Ok(())` return would trigger Clippy warnings about unnecessary Result wrapping.

Learnt from: generall
PR: qdrant/qdrant#6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The `populate()` method in `QuantizedMmapStorage` does not return a `Result` because the underlying `mmap.populate()` method doesn't return a Result either. Adding an unnecessary `Ok(())` would cause Clippy to complain.

lib/gridstore/src/page.rs (3)

Learnt from: generall
PR: #6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The populate() method in QuantizedMmapStorage correctly returns void instead of a Result because it directly implements the Madviseable trait which defines populate(&self) without a return type. Adding an unnecessary Ok(()) return would trigger Clippy warnings about unnecessary Result wrapping.

Learnt from: generall
PR: #6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The populate() method in QuantizedMmapStorage does not return a Result because the underlying mmap.populate() method doesn't return a Result either. Adding an unnecessary Ok(()) would cause Clippy to complain.

Learnt from: coszio
PR: #6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.

lib/common/memory/src/chunked_utils.rs (3)

Learnt from: generall
PR: #6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The populate() method in QuantizedMmapStorage correctly returns void instead of a Result because it directly implements the Madviseable trait which defines populate(&self) without a return type. Adding an unnecessary Ok(()) return would trigger Clippy warnings about unnecessary Result wrapping.

Learnt from: generall
PR: #6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The populate() method in QuantizedMmapStorage does not return a Result because the underlying mmap.populate() method doesn't return a Result either. Adding an unnecessary Ok(()) would cause Clippy to complain.

Learnt from: coszio
PR: #6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.

lib/segment/src/vector_storage/dense/mmap_dense_vectors.rs (4)

Learnt from: generall
PR: #6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The populate() method in QuantizedMmapStorage correctly returns void instead of a Result because it directly implements the Madviseable trait which defines populate(&self) without a return type. Adding an unnecessary Ok(()) return would trigger Clippy warnings about unnecessary Result wrapping.

Learnt from: generall
PR: #6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The populate() method in QuantizedMmapStorage does not return a Result because the underlying mmap.populate() method doesn't return a Result either. Adding an unnecessary Ok(()) would cause Clippy to complain.

Learnt from: coszio
PR: #6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.

Learnt from: timvisee
PR: #6569
File: lib/segment/src/vector_storage/sparse/volatile_sparse_vector_storage.rs:56-76
Timestamp: 2025-05-21T12:52:15.229Z
Learning: In volatile sparse vector storage implementations, the total_sparse_size field should be updated when vectors are inserted, updated, or deleted to maintain an accurate count of the total number of non-zero elements across all vectors.

lib/common/memory/src/mmap_type_readonly.rs (3)

Learnt from: generall
PR: #6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The populate() method in QuantizedMmapStorage correctly returns void instead of a Result because it directly implements the Madviseable trait which defines populate(&self) without a return type. Adding an unnecessary Ok(()) return would trigger Clippy warnings about unnecessary Result wrapping.

Learnt from: generall
PR: #6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The populate() method in QuantizedMmapStorage does not return a Result because the underlying mmap.populate() method doesn't return a Result either. Adding an unnecessary Ok(()) would cause Clippy to complain.

Learnt from: timvisee
PR: #6644
File: lib/gridstore/src/blob.rs:41-57
Timestamp: 2025-06-06T07:52:38.478Z
Learning: In the Blob trait implementations for gridstore (lib/gridstore/src/blob.rs), the maintainer prefers using assert! for input validation over Result-based error handling, as they don't expect invalid inputs in real scenarios and changing to Result would require updating the entire trait interface.

🧰 Additional context used

🧠 Learnings (5)

📓 Common learnings

Learnt from: generall
PR: qdrant/qdrant#6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The `populate()` method in `QuantizedMmapStorage` correctly returns void instead of a `Result` because it directly implements the `Madviseable` trait which defines `populate(&self)` without a return type. Adding an unnecessary `Ok(())` return would trigger Clippy warnings about unnecessary Result wrapping.

Learnt from: generall
PR: qdrant/qdrant#6323
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:12-16
Timestamp: 2025-04-07T23:31:22.515Z
Learning: The `populate()` method in `QuantizedMmapStorage` does not return a `Result` because the underlying `mmap.populate()` method doesn't return a Result either. Adding an unnecessary `Ok(())` would cause Clippy to complain.