Depends on #6444.

Add in-memory payload index for full text using mmap as storage.

Loading the in-memory full text payload index from mmap is 33x faster than from RocksDB, as shown in this simple benchmark:

• bfb --text-payloads --text-payload-length 40 -n 500000 -d1 --indexing-threshold 100:

  • Old RocksDB based index: 2.4s, 2.5s, 2.5s, 2.6s = avg 2.5s
  • New mmap based index: 77.2ms, 75.1ms, 77.2ms, 74.7ms = avg 0.076s (33x faster!)

• bfb --text-payloads --text-payload-length 100 --text-payload-vocabulary 50000 -n 500000 -d1 --indexing-threshold 100:

  • Old RocksDB based index: 6.6s, 6.6s, 6.7s, 6.6s = avg 6.6s
  • New mmap based index: 202.8ms, 200.6ms, 211.2ms, 201.8ms = avg 0.2s (33x faster!)

Tasks

• Switch point_to_tokens_count key from option to number (will be done separately)
• Add test
• Do load benchmark
• Drop mmap caches after load
• Merge Add in-memory payload indices on mmap storage (numeric, map) #6444
• Rebase on dev
• Undraft

All Submissions:

• Contributions should target the dev branch. Did you create your branch from dev?
• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

1. Does your submission pass tests?
2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your core changes, as applicable?
• Have you successfully ran tests with your changes locally?