Skip to content

Conversation

alexanderguzhva
Copy link
Contributor

@alexanderguzhva alexanderguzhva commented Feb 19, 2025

This PR introduces a backport of a combination of zilliztech/knowhere#996 and zilliztech/knowhere#1032 that allow to have memory-mapped and zerocopy indces.

The root underlying idea is that we replace certain std::vector<> containers with a custom faiss::MaybeOwnedVector<> container, which may behave either as std::vector<>, or as a view of a certain pointer / descriptor. We don't replace all the instances of std::vector<>, but the largest ones.

This change affects IndexFlatCodes-based and IndexHNSW CPU indices.

(done) alter IVF lists as well.
(done) alter binary indices as well.

Memory-mapped index works like this:

std::unique_ptr<faiss::Index> index_mm(
            faiss::read_index(filenamename.c_str(), faiss::IO_FLAG_MMAP_IFC));

In theory, it should be ready to be used from Python. All the descriptor management should be working.

Zero-copy index works like this:

#include <faiss/impl/zerocopy_io.h>

faiss::ZeroCopyIOReader reader(buffer.data(), buffer.size());
std::unique_ptr<faiss::Index> index_zc(faiss::read_index(&reader));

All the pointer management for faiss::ZeroCopyIOReader should be handled manually.
I'm not sure how to plug this into Python yet, maybe, some ref-counting is required.

(done) some refactoring

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
@alexanderguzhva
Copy link
Contributor Author

alexanderguzhva commented Feb 20, 2025

@mdouze Would please be able to suggest a solution for the following problem. Because I changed the container type for IndexFlatCodes::codes field, it triggered some errors in python unit tests, which are related to these .codes (bcz the field type got changed). What would be the most meaningful solution for dealing with it? Basically, I'm not sure how to minimize the amount of backward-compatibility problems for the code that uses Faiss across the world, as I am not an expert swig user

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Copy link
Contributor

@mdouze mdouze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this. See a few inline reviews.
As long as the interface of MaybeOwnedVector is sufficiently close of std::vector, it should work seamlessly in Python (duck typing). I'll see if some adaptations to the swig interface are required.
It is not clear to me why the MappedFileIOReader is needed rather than the regular FileIOReader. Is it to maintain the ownership of the reader within the MaybeOwnedVector and close the file when deallocated?

impl/pq4_fast_scan.cpp
impl/pq4_fast_scan_search_1.cpp
impl/pq4_fast_scan_search_qbs.cpp
impl/residual_quantizer_encode_steps.cpp
impl/io.cpp
impl/lattice_Zn.cpp
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do these two file disappear?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicates

READVECTOR(target);
}

template <typename VectorT>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possible to factorize with previous function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a good suggestion, let me see...

}

template <typename VectorT>
void read_vector(VectorT& target, IOReader* f) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to enable this via an IO flag ? The IVF memory mapped files work on regular FileIOReader

https://github.com/facebookresearch/faiss/blob/main/faiss/invlists/OnDiskInvertedLists.cpp#L771

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

> OnDiskInvertedLists is unsupported on Windows.

my code allows mmap functionality on Windows as well. Let me take a look

@facebook-github-bot
Copy link
Contributor

@mdouze has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@mdouze
Copy link
Contributor

mdouze commented Feb 21, 2025

This patch should fix the python test:
https://gist.github.com/mdouze/83eff60f99ae1a3cba524c6b8320dd8f
could you integrate it?

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
@alexanderguzhva
Copy link
Contributor Author

alexanderguzhva commented Feb 21, 2025

It is not clear to me why the MappedFileIOReader is needed rather than the regular FileIOReader.

in order to have a correct support for Windows

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
@facebook-github-bot
Copy link
Contributor

@mdouze has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
@alexanderguzhva
Copy link
Contributor Author

@mdouze I'm going to add IVFs and maybe Binary indices today as well, so please don't merge it yet :)

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
@alexanderguzhva alexanderguzhva changed the title [WIP] Memory-mapping and Zero-copy deserializers Memory-mapping and Zero-copy deserializers Feb 24, 2025
@alexanderguzhva
Copy link
Contributor Author

@mdouze please let me know which other unit tests are useful to have
I've added the code for ArrayInvertedLists and binary vectors as well.

@alexanderguzhva
Copy link
Contributor Author

the PR is ready to be reviewed

@facebook-github-bot
Copy link
Contributor

@mdouze has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@mdouze merged this pull request in 55a3c2a.

@facebook-github-bot
Copy link
Contributor

This pull request has been reverted by 15491a1.

@mdouze mdouze mentioned this pull request Mar 21, 2025
mdouze added a commit to mdouze/faiss that referenced this pull request Mar 21, 2025
Summary:
This is an attempt to re-land the diff stack D69972250  D70982449

It was reverted because the bottom of the stack did not pass the tests.

The original code comes from Alexandr Guzhva's  facebookresearch#4199

Differential Revision: D71614511
mdouze added a commit to mdouze/faiss that referenced this pull request Mar 21, 2025
Summary:

This is an attempt to re-land the diff stack D69972250  D70982449

It was reverted because the bottom of the stack did not pass the tests.

The original code comes from Alexandr Guzhva's  facebookresearch#4199

Differential Revision: D71614511
mdouze added a commit to mdouze/faiss that referenced this pull request Mar 21, 2025
Summary:
Pull Request resolved: facebookresearch#4250

This is an attempt to re-land the diff stack D69972250  D70982449

It was reverted because the bottom of the stack did not pass the tests.

The original code comes from Alexandr Guzhva's  facebookresearch#4199

Differential Revision: D71614511
mdouze added a commit to mdouze/faiss that referenced this pull request Mar 21, 2025
Summary:

This is an attempt to re-land the diff stack D69972250  D70982449

It was reverted because the bottom of the stack did not pass the tests.

The original code comes from Alexandr Guzhva's  facebookresearch#4199

Differential Revision: D71614511
mdouze added a commit to mdouze/faiss that referenced this pull request Mar 21, 2025
Summary:

This is an attempt to re-land the diff stack D69972250  D70982449

It was reverted because the bottom of the stack did not pass the tests.

The original code comes from Alexandr Guzhva's  facebookresearch#4199

To the adsmarket steward: the diff was already accepted by your team (see D70982449), but reverted for an independent reason. So should be easy to accept now.

Reviewed By: mengdilin

Differential Revision: D71614511
mdouze added a commit to mdouze/faiss that referenced this pull request Mar 24, 2025
Summary:

This is an attempt to re-land the diff stack D69972250  D70982449

It was reverted because the bottom of the stack did not pass the tests.

The original code comes from Alexandr Guzhva's  facebookresearch#4199

To the adsmarket steward: the diff was already accepted by your team (see D70982449), but reverted for an independent reason. So should be easy to accept now.

Reviewed By: mengdilin

Differential Revision: D71614511
facebook-github-bot pushed a commit that referenced this pull request Mar 24, 2025
Summary:
Pull Request resolved: #4250

This is an attempt to re-land the diff stack D69972250  D70982449

It was reverted because the bottom of the stack did not pass the tests.

The original code comes from Alexandr Guzhva's  #4199

To the adsmarket steward: the diff was already accepted by your team (see D70982449), but reverted for an independent reason. So should be easy to accept now.

Reviewed By: mengdilin

Differential Revision: D71614511

fbshipit-source-id: 94139b4a4d457afe0d37ac95342537414aa81e7a
abhinavdangeti added a commit to blevesearch/faiss that referenced this pull request May 20, 2025
…iss@bleve` (#52)

Merge results:
```
|\
* ca874b6 Abhinav Dangeti | Fix type mismatches within unit test: TEST(TestHamming, test_hamming_knn)
* e255b9b Abhinav Dangeti | Adapt signature change of `get_InvertedListScanner` in faiss/IndexIVFPQ.cpp
* 90fe29b Abhinav Dangeti | Remove redundant cmake install over target `faiss_c`
*   0882dd3 Abhi Dangeti | Merge branch 'bleve' into main_1.11.0
|\
| * 0be294a Deepkaran Salooja | Implement compute_distance_to_codes_for_list and compute_distance_table for IndexIVFPQ (#50)
| * 352484e Rahul Rampure | MB-65473: Batch converter for vector to cluster IDs (#49)
| * 14a4a60 Rahul Rampure | MB-65473: Refactor and Optimize Pre-Filtered Vector Search (#48)
| *   b4cc942 Abhi Dangeti | MB-65243: Merge 'facebookresearch/faiss@v1.10.0' into 'blevesearch/faiss@bleve' (#46)
| |\
| | * 8d33b5c Abhinav Dangeti | MB-65243: Merge 'facebookresearch/faiss@v1.10.0' into 'blevesearch/faiss@bleve'
| |/|
| * | 8eecdb6 Rahul Rampure | MB-63643: Fix missing num_threads clauses (#44)
| * | 224acef Deepkaran Salooja | MB-61093 Fix memory leak for SQDistanceComputer (#43)
| * | 3001b51 Deepkaran Salooja | MB-61093 Add method to compute distance from codes for IVF index (#41)
| * | b747c55 Aditi Ahuja | MB-62230 - Updated closest_centroids API to include params (#38)
| * | 26d9b35 Aditi Ahuja | MB-62230 - Extended c_api to search only specified clusters with params. (#35)
| * | f077bf9 Abhi Dangeti | Build libfaiss with AVX2 support when requested, rather than libfaiss (#37)
| * |   5ab1ce0 Abhi Dangeti | MB-62577: Merge 'facebookresearch/faiss@v1.8.0' into blevesearch/faiss@bleve
| |\ \
| | * | 3306e58 Abhinav Dangeti | MB-62577: Merge 'facebookresearch/faiss@v1.8.0' into blevesearch/faiss@bleve
| |/| |
| * | | d9db66a Rahul Rampure | MB-62221: API to free a buffer allocated in C runtime (#30)
| * | | a2f4183 Rahul Rampure | MB-62221: Fix buffer overflow (#29)
| * | | 7977457 Rahul Rampure | MB-61930: Add a num_threads clause to every openMP pragma. (#25)
| * | | a30eaa2 Rahul Rampure | MB-61930: Optimize Thread Management in High Throughput Scenarios (#24)
| * | | 2ce3883 Thejas-bhat | MB-59575: Revert memcpy optimizations for flat indexes (#23)
| * | | 7c3c7d1 Thejas-bhat | MB-59575: Refactor member variables alignment of IndexFlatCodes (#22)
| * | | 17c3992 Thejas-bhat | MB-59575: Reducing copy overhead of already memory mapped content (#17)
| * | | 38f6b60 Chris Hillery | Fix build on Windows (#21)
| * | | 4143984 SaptarshiSen-CB | MB-61609: Fix zero sa_code_size (#19)
| * | | 7b119f4 Rahul Rampure | MB-60739: Fix integer overflow (#15)
| * | | 6851683 Rahul Rampure | MB-60657: Fix integer overflow (#14)
| * | | 8672bf3 Thejas-bhat | Size API to get the index's size (#13)
| * | | b34ccf6 Aditi Ahuja | MB-60202 - IDMap2 Selector (#12)
| * | | a623ec6 Thejas-bhat | Introducing a new reader to read index using a pointer (#8)
| * | | 4dd26f8 Chris Hillery | Add INSTALL() directive for faiss_c (#7)
| * | | 14fd16a Chris Hillery | Suppress (thousands of) warnings when building with GCC (#6)
| * | | 44febf0 Abhi Dangeti | Address incorrect import within c_api/IndexIVF_c_ex.cpp (#5)
| * | | 1b295e4 Abhi Dangeti | Add build instructions for IndexIVF_c_ex.cpp and Index_c_ex.cpp (#4)
| * | | 334021a Abhi Dangeti | additional index APIs (#3)
| * | | f0bbc06 Abhi Dangeti | Introducing index IO operations over char buffer (#2)
* | | | ea1cdf0 Michael Norris | Increment next release, v1.11.0 (facebookresearch#4308)
* | | | 70c4537 simshi | fix: algorithm of spreading vectors over shards (facebookresearch#4299)
* | | | d4fa401 Michael Norris | Add RaBitQ to the swigfaiss so we can access its properties correctly in python (facebookresearch#4304)
* | | | c75f166 Satyendra Mishra | Add date and time to the codec file path so that the file doesn't get overridden with each run (facebookresearch#4303)
* | | | a3cd63f Aditya Vidyadhar Kamath | Skip mmap test case in AIX. (facebookresearch#4275)
* | | | e36897f Michael Norris | Fix overflow of int32 in IndexNSG (facebookresearch#4297)
* | | | 117aafd Michael Simpson | Fix Type Error in Conditional Logic (facebookresearch#4294)
* | | | 928333c Jim Meyering | faiss/gpu/GpuAutoTune.cpp: fix llvm-19-exposed -Wunused-but-set-variable warnings
* | | | bb04bf6 Bhavik Sheth | Add missing header in faiss/CMakeLists.txt (facebookresearch#4285)
* | | | d9cfd00 Satyendra Mishra | Implement is_spherical and normalize_L2 booleans as part of the training APIs (facebookresearch#4279)
* | | | 915f719 Michael Norris | Fix nightly by pinning conda-build to prevent regression in 25.3.2 (facebookresearch#4287)
* | | | de5e85e generatedunixname89002005287564 | Fix CQS signal. Id] 88153895 -- readability-redundant-string-init in fbcode/faiss (facebookresearch#4283)
* | | | 7eac034 Satyendra Mishra | Add normalize_l2 boolean to distributed training API
* | | | 0dfb599 Jaap Aarts | Handle insufficient driver gracefully (facebookresearch#4271)
* | | | d4e236b Alexandr Guzhva | relax input params for IndexIVFRaBitQ::get_InvertedListScanner() (facebookresearch#4270)
* | | | df9e2c4 Alexandr Guzhva | Fix a placeholder for 'unimplemented' in mapped_io.cpp (facebookresearch#4268)
* | | | 0d3aff9 wwq | fix bug: IVFPQ of raft/cuvs does not require redundant check (facebookresearch#4241)
* | | | a4401c1 Kaival Parikh | Allow using custom index readers and writers (facebookresearch#4180)
* | | | 636d95e Tarang Jain | Upgrade to libcuvs=25.04 (facebookresearch#4164)
* | | | 7f523f0 Junjie Qi | ignore regex (facebookresearch#4264)
* | | | ccc2b33 Alexandr Guzhva | fix a serialization problem in RaBitQ (facebookresearch#4261)
* | | | 13255a8 Kaival Parikh | Publish the C API to Conda (facebookresearch#4186)
* | | | 3a49130 Alexandr Guzhva | RaBitQ implementation (facebookresearch#4235)
* | | | c2fc549 Satyendra Mishra | Pass row filters to Hive Reader to filter rows (facebookresearch#4256)
* | | | 6116d36 Mayank Bhatia | Grammar fix in FlatIndexHNSW (facebookresearch#4253)
* | | | 1debb7d Matthijs Douze | re-land mmap diff (facebookresearch#4250)
* | | | 0f2035c Richard Barnes | Fix CUDA kernel index data type in faiss/gpu/impl/DistanceUtils.cuh +10 (facebookresearch#4246)
* | | | 1dcbb4a Alexandr Guzhva | fix `IVFPQFastScan::RangeSearch()` on the `ARM` architecture (facebookresearch#4247)
* | | | 8bce244 Mengdi Lin | fix integer overflow issue when calculating imbalance_factor (facebookresearch#4245)
* | | | 5adab67 Rohil Shah | Fix bug with metric_arg in IndexHNSW (facebookresearch#4239)
* | | | f2f7a66 Mengdi Lin | Back out "test merge with internal repo" (facebookresearch#4244)
* | | | caa5f24 Junjie Qi | test merge with internal repo (facebookresearch#4242)
* | | | 9e808d4 Richard Barnes | Remove unused exception parameter from faiss/impl/ResultHandler.h (facebookresearch#4243)
* | | | fec7ce9 Gustav von Zitzewitz | SearchParameters support for IndexBinaryFlat (facebookresearch#4055)
* | | | df6a8f6 George Wang | Address compile errors and warnings (facebookresearch#4238)
* | | | 15491a1 Saumya Agarwal | Revert D69972250: Memory-mapping and Zero-copy deserializers
* | | | fbc7db2 Saumya Agarwal | Revert D69984379: mem mapping and zero-copy python fixes
* | | | 631b0fd Matthijs Douze | mem mapping and zero-copy python fixes (facebookresearch#4212)
* | | | 55a3c2a Alexandr Guzhva | Memory-mapping and Zero-copy deserializers (facebookresearch#4199)
* | | | 653be59 Richard Barnes | Use `nullptr` in faiss/gpu/StandardGpuResources.cpp (facebookresearch#4232)
* | | | 3d96ad5 Lucian Grijincu | faiss: fix non-templated hammings function (facebookresearch#4195)
* | | | 4cd2f6e Junjie Qi | Support non-partition col and map in the embedding reader (facebookresearch#4229)
* | | | a22ec32 Junjie Qi | Support cosine distance for training vectors (facebookresearch#4227)
* | | | c109174 Richard Barnes | Fix LLVM-19 compilation issue in faiss/AutoTune.cpp (facebookresearch#4220)
* | | | 615c17e Shuyao Qi | Add missing #include in code_distance-sve.h (facebookresearch#4219)
* | | | eab52af Tom Jackson | Fix cloning and reverse index factory for NSG indices (facebookresearch#4151)
* | | | 1a295cd George Wang | Remove python_abi to fix nightly (facebookresearch#4217)
* | | | 4cea80b Shuyao Qi | Make static method in header inline (facebookresearch#4214)
* | | | 835b3ea Michael Norris | Fix IVF quantizer centroid sharding so IDs are generated (facebookresearch#4197)
* | | | 65222b3 Michael Norris | Pin lief to fix nightly (facebookresearch#4211)
* | | | 7cb4556 lkuffo | Fix Sapphire Rapids never loading in Python bindings (facebookresearch#4209)
* | | | 20c7ca3 Michael Norris | Upgrade openblas to 0.3.29 for ARM architectures (facebookresearch#4203)
* | | | 55d022f George Wang | Attempt to nightly fix (facebookresearch#4204)
* | | | 00ce0e2 Navneet Verma | Add the support for IndexIDMap with Cagra index (facebookresearch#4188)
* | | | 1fe8b8b Nicolas De Carli | Remove unused variable (facebookresearch#4205)
* | | | 6b65289 Divye Gala | Pass `store_dataset` argument along to cuVS CAGRA (facebookresearch#4173)
* | | | d72d0ca Michael Norris | Fix nightly by installing earlier version of lief (facebookresearch#4198)
* | | | 657c563 Bhavik Sheth | Add bounds checking to hnsw nb_neighbors (facebookresearch#4185)
* | | | f0e3832 George Wang | Check for not completed
* | | | aff6bfc Michael Norris | Add sharding convenience function for IVF indexes (facebookresearch#4150)
* | | | 1d8f393 Kaival Parikh | Handle plain SearchParameters in HNSW searches (facebookresearch#4167)
* | | | c6adc01 Michael Norris | Update INSTALL.md to remove some raft references, add missing dependency (facebookresearch#4176)
* | | | 95955d8 Kota Yamaguchi | Fix install error when building avx512_spr variant (facebookresearch#4170)
* | | | d720155 Amir Sadoughi | Update README.md (facebookresearch#4169)
* | | | 9896beb simshi | fix: gpu tests link failure with static lib (facebookresearch#4137)
* | | | 6c04699 Mulugeta Mammo | Fix the order of parameters in bench_scalar_quantizer_distance. (facebookresearch#4159)
* | | | 3ec2fbd Tarang Jain | Update CAGRA docs (facebookresearch#4152)
* | | | 6718dae Kaival Parikh | Expose IDSelectorBitmap in the C_API (facebookresearch#4158)
* | | | 9bc4b67 Jesper Stemann Andersen | Added support for building for MinGW, in addition to MSVC (facebookresearch#4145)
| |_|/
|/| |
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants