feat: support prefilter #247
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
86a804b to
9edd281
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR introduces support for prefiltering by renaming the configuration option from prererank_filtering to prefilter and updating related code throughout the codebase. It also optimizes query/insert operations by eliminating redundant matrix multiplications and refactors insertion logic to use new helper functions.
- Renames the configuration setting and updates its usage across modules.
- Adds a new filter method to the SearchFetcher trait and updates its implementations.
- Refactors vector insertion logic and adjusts test workflows to cover pg16 and pg17 features.
Reviewed Changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/general/*.slt | Added DROP TABLE cleanup commands in test setups |
| src/index/scanners/mod.rs | Adds new filter method to SearchFetcher trait |
| src/index/scanners/maxsim.rs | Refactors vector handling with clearer naming and cloning |
| src/index/scanners/default.rs | Introduces rerank_wrapper and uses seq_filter for prefilter |
| src/index/gucs.rs, src/index/am/mod.rs | Adjusts configuration accesses from prererank_filtering to prefilter |
| src/index/algorithm.rs, crates/algorithm/* | Updates insertion logic to use new helper functions, improves prefetcher traits |
| .github/workflows/check.yml | Updates test file patterns and adds version-specific test jobs |
Comments suppressed due to low confidence (2)
.github/workflows/check.yml:176
- [nitpick] Please verify that the updated test file glob pattern covers all intended test cases and that no tests are unintentionally omitted from the CI runs.
sqllogictest --db $USER --user $USER './tests/general/*.slt'
src/index/scanners/default.rs:133
- Using BinaryHeap::from(results) for filtering could incur additional overhead if the results are large; verify that this conversion meets performance targets under heavy load.
let seq = seq_filter(BinaryHeap::from(results), |key| {
usamoi
requested changes
May 12, 2025
73484de to
a6f5179
Compare
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai>
usamoi
requested changes
May 13, 2025
fa56eae to
7ee8814
Compare
usamoi
reviewed
May 14, 2025
src/index/scanners/default.rs
Outdated
| )) | ||
| } | ||
| // 强制prefilter 检查还会不会出错 | ||
| // TODO 记得改回来 |
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai>
usamoi
approved these changes
May 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Feat
rerank_in_table=truewithread_stream(pg17) andprefetch_buffer(not pg17)Optimization
Chore
vchordrq.prererank_filteringintovchordrq.prefilterTest