Skip to content

Add mutable map payload index on top of Gridstore #6634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Jun 5, 2025
Merged

Add mutable map payload index on top of Gridstore #6634

merged 8 commits into from
Jun 5, 2025

Conversation

timvisee
Copy link
Member

@timvisee timvisee commented Jun 3, 2025 •
edited
Loading

Depends on #6609

Similar to #6609, this implements Gridstore as alternative storage backend for a payload index. This implements it for the map index.

Tasks

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

@timvisee timvisee added this to the Remove RocksDB milestone Jun 3, 2025
@timvisee
Copy link
Member Author

timvisee commented Jun 3, 2025 •
edited
Loading

Here's a quick and dirty test with the following setup:

bfb -d 4 --int-payloads 100 --max-int-payloads 10 --indexing-threshold 0 -n 1000000

On RocksDB:

  • Cold load: 1.64s
  • Hot load: 1.59s
  • Upsert time: 18s
  • Disk usage: 13M + 13M = 25M

On Gridstore:

  • Cold load: 1.27s
  • Hot load: 0.55s
  • Upsert time: 5s
  • Disk usage: 36M + 36M = 71M

The Gridstore implementation currently consumes more space because the configured Gridstore options aren't ideal for very small numeric values. This we can tune later.

@timvisee timvisee force-pushed the mutable-map-index-gridstore branch 2 times, most recently from a62b7cf to 7d31d7b Compare June 4, 2025 11:37
@timvisee timvisee force-pushed the mutable-numeric-index-gridstore branch from 067cbf5 to c7d0b2b Compare June 4, 2025 15:47
@timvisee timvisee force-pushed the mutable-map-index-gridstore branch from 7d31d7b to 0972557 Compare June 4, 2025 15:50
@timvisee timvisee mentioned this pull request Jun 4, 2025
Merged
12 tasks
Base automatically changed from mutable-numeric-index-gridstore to dev June 5, 2025 07:10
@timvisee timvisee force-pushed the mutable-map-index-gridstore branch from 0972557 to 6488bdd Compare June 5, 2025 07:11
@timvisee timvisee marked this pull request as ready for review June 5, 2025 07:12
@timvisee timvisee requested a review from Copilot June 5, 2025 07:12
Copilot

This comment was marked as resolved.

Sign in to view

@coderabbitai coderabbitai

This comment was marked as resolved.

Sign in to view
coderabbitai[bot]

This comment was marked as resolved.

Sign in to view

coderabbitai[bot]

This comment was marked as resolved.

Sign in to view

@timvisee timvisee merged commit 4d71f06 into dev Jun 5, 2025
17 checks passed
@timvisee timvisee deleted the mutable-map-index-gridstore branch June 5, 2025 07:54
@coderabbitai coderabbitai bot mentioned this pull request Jun 6, 2025
Merged
12 tasks
generall pushed a commit that referenced this pull request Jul 17, 2025
@timvisee @generall
Add mutable map payload index on top of Gridstore (#6634)
705607f 
* Serialize vector of ecow strings using cbor

* Adjust trait bounds for Gridstore support in map index

* Add Gridstore backend to mutable map index

* Add Gridstore builder

* Enable new map index in index selector

* Add tests

* Use different Gridstore block size depending on map data type

* Fix review remarks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

Copilot code review Copilot Copilot left review comments

@generall generall generall approved these changes

@JojiiOfficial JojiiOfficial JojiiOfficial approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Remove RocksDB
Development

Successfully merging this pull request may close these issues.
3 participants
@timvisee @generall @JojiiOfficial