Skip to content

Conversation

timvisee
Copy link
Member

@timvisee timvisee commented May 7, 2025

Depends on #6495.

Add in-memory payload index for geo points using mmap as storage.

Loading the in-memory full text payload index from mmap is 24x faster than from RocksDB, as shown in this simple benchmark:

  • bfb --geo-payloads -n 2000000 -d1 --indexing-threshold 100:
    • Old RocksDB based index: 11.78s, 11.86s, 11.57s, 11.66s = avg 11.72s
    • New mmap based index: 0.489s, 0.476s, 0.476s, 0.477s = avg 0.480ms (24x faster)

Tasks

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

@timvisee timvisee changed the title Payload index ram on mmap geo Add in-memory payload index on mmap storage (geo) May 7, 2025
@timvisee timvisee force-pushed the payload-index-ram-on-mmap-full-text branch from f7ed774 to 08244e9 Compare May 8, 2025 10:35
@timvisee timvisee force-pushed the payload-index-ram-on-mmap-geo branch 2 times, most recently from 17294ad to b6d6d10 Compare May 9, 2025 14:17
Base automatically changed from payload-index-ram-on-mmap-full-text to dev May 12, 2025 08:58
@timvisee timvisee force-pushed the payload-index-ram-on-mmap-geo branch from b6d6d10 to 2299219 Compare May 12, 2025 09:17
@timvisee timvisee marked this pull request as ready for review May 12, 2025 12:34
Copilot

This comment was marked as resolved.

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

@generall generall merged commit 932d953 into dev May 13, 2025
17 checks passed
@generall generall deleted the payload-index-ram-on-mmap-geo branch May 13, 2025 11:06
n0x29a pushed a commit that referenced this pull request May 14, 2025
* Add storage enum to geo index

* Construct immutable geo index from mmap geo index

* Load immutable geo index from mmap storage if on-disk is false

* When immutable geo index is loaded into memory, clear mmap cache

* Adjust immutable geo index from mmap loading, consider point deletions

* Rename get_db_wrapper to db_wrapper

* Add immutable mmap geo index test, currently still failing

* Properly load geo index

* Add congruence test

* Also test with deletions

* Fix clipping warnings
generall pushed a commit that referenced this pull request May 22, 2025
* Add storage enum to geo index

* Construct immutable geo index from mmap geo index

* Load immutable geo index from mmap storage if on-disk is false

* When immutable geo index is loaded into memory, clear mmap cache

* Adjust immutable geo index from mmap loading, consider point deletions

* Rename get_db_wrapper to db_wrapper

* Add immutable mmap geo index test, currently still failing

* Properly load geo index

* Add congruence test

* Also test with deletions

* Fix clipping warnings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants