feat: pinning index in memory when building, second try #181
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Let's try it on 100M dataset and see how flamegraph goes |
|
2c4t, on laion-100m:
next steps:
cc @VoVAllen |
Signed-off-by: usamoi <usamoi@outlook.com>
|
Why do we need to merge two segments together? The optimization result is pretty close to the flamegraph result. |
|
And what is 2c4t? |
4 vcpu
not necessary; helpful to make it optional |
bc95918 to
dd76752
Compare
Signed-off-by: usamoi <usamoi@outlook.com>
|
i4i.xlarge, laion-100m:
|
There was a problem hiding this comment.
Pull Request Overview
This PR implements a "pinning index in memory" feature by introducing a new cache function and updating related build and index option types. Key changes include:
- Adding a new cache function in crates/algorithm/src/cache.rs that builds an index trace.
- Defining new indexing and build options in src/index/types.rs and refactoring related options in crates/algorithm/src/types.rs.
- Adding a clone_into_boxed method in src/index/storage.rs to safely duplicate PostgresPage data using unsafe pointer copying.
Reviewed Changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| crates/algorithm/src/cache.rs | New cache function implementation that reads index data and builds a trace. |
| src/index/types.rs | New type definitions for internal and external build options to support the pinned index feature. |
| src/index/storage.rs | Adds a Debug derive and a clone_into_boxed method for PostgresPage to facilitate boxed cloning. |
| crates/algorithm/src/lib.rs | Integrates the new cache module and updates module exports. |
| crates/algorithm/src/types.rs | Refactors index options by removing build option types and using a simplified VchordrqIndexOptions struct. |
| crates/algorithm/src/build.rs | Updates the build function parameter type to use the new VchordrqIndexOptions. |
Comments suppressed due to low confidence (1)
src/index/storage.rs:46
- Ensure that the unsafe pointer copy in clone_into_boxed is safe for all fields of PostgresPage, particularly if the structure evolves to include non-POD types.
pub fn clone_into_boxed(&self) -> Box<Self> {
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
issue: #117
first try: #150