MRG: implement manifest retrieval from Rust via FFI for RevIndex
#3630
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## latest #3630 +/- ##
==========================================
- Coverage 88.44% 88.33% -0.12%
==========================================
Files 136 137 +1
Lines 23401 23467 +66
Branches 2269 2270 +1
==========================================
+ Hits 20697 20729 +32
- Misses 2394 2428 +34
Partials 310 310
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@luizirber any commentary/suggestions on the code I rescued, and in particular on |
|
(There's a seg fault in there that I don't quite understand, too :) |
RevIndexRevIndex
|
Ready for review @luizirber @bluegenes ! |
There was a problem hiding this comment.
Main question: the goal here is to modify a copy of the revindex Manifest, and potentially replace it, NOT to edit in place?
I think that's fine, and most manifest operations are not that demanding, but might be quite a bit of memory to create two copies (one in Rust, one in Python) for large ones like the SRA metagenomes.
(and probably the best we can do at the moment, without thinking deeper into how ownership works across Python/Rust...)
yes to all of the above 😆 . More will come, but this is minimally necessary for decent RocksDB inspection and picklisting... |
# sourmash v4.9.0 release notes This release adds two significant feature sets to sourmash, without introducing any breaking changes. First, sourmash now fully supports fast, low-memory disk-based inverted indexes based on RocksDB. This functionality has been part of [the branchwater plugin](https://github.com/sourmash-bio/sourmash_plugin_branchwater) for a while, but it is now accessible via the sourmash command line and Python API. Second, we have added skip-mer sketching to sourmash, joining DNA, protein, dayhoff, and hp encodings. Skip-mers allow more mismatches than DNA k-mers and can be useful when comparing fast-evolving sequences such as virus and phage genomes. Documentation for the RocksDB indexes and skip-mer encodings is available in the [command-line docs](https://sourmash.readthedocs.io/en/latest/command-line.html). Major new features: * Fully support fast, low-memory RocksDB indexes in Python (#3545) * Fully support skip-mers at the Python level; provide documentation (#3627) * Remove support for python 3.10 (#3606) Cleanup and documentation updates: * add default to `add_scaled_arg` in Python CLI utils (#3609) * use `match/case` in `sourmash index` implementation (#3604) * use single quotes inside sqlite statements (#3556) Developer updates: * implement manifest retrieval from Rust via FFI for `RevIndex` (#3630) * make the RocksDB handle directly accessible to external code (#3468) * fix linear gather in Rust (#3605) * fix beta clippy errors (#3548) * fix deprecations (#3613) * update Makefile with 'offline', 'wheel' (#3579) * update ubuntu image version for CI (#3623) * Minhash deserialize hashfunction errorhandling (#3560) Automated updates: * Bump DeterminateSystems/nix-installer-action from 16 to 17 (#3626) * Bump getset from 0.1.4 to 0.1.5 (#3567) * Bump histogram from 0.11.2 to 0.11.3 (#3574) * Bump log from 0.4.25 to 0.4.26 (#3549) * Bump log from 0.4.26 to 0.4.27 (#3587) * Bump needletail from 0.6.1 to 0.6.3 (#3553) * Bump prefix-dev/setup-pixi from 0.8.1 to 0.8.2 (#3538) * Bump prefix-dev/setup-pixi from 0.8.2 to 0.8.3 (#3551) * Bump prefix-dev/setup-pixi from 0.8.3 to 0.8.4 (#3602) * Bump prefix-dev/setup-pixi from 0.8.4 to 0.8.7 (#3616) * Bump prefix-dev/setup-pixi from 0.8.7 to 0.8.8 (#3621) * Bump pypa/cibuildwheel from 2.22.0 to 2.23.0 (#3564) * Bump pypa/cibuildwheel from 2.23.0 to 2.23.1 (#3581) * Bump pypa/cibuildwheel from 2.23.1 to 2.23.2 (#3603) * Bump pypa/cibuildwheel from 2.23.2 to 2.23.3 (#3625) * Bump rand from 0.9.0 to 0.9.1 (#3620) * Bump roaring from 0.10.10 to 0.10.12 (#3608) * Bump serde from 1.0.217 to 1.0.218 (#3550) * Bump serde from 1.0.218 to 1.0.219 (#3576) * Bump serde_json from 1.0.138 to 1.0.139 (#3552) * Bump serde_json from 1.0.139 to 1.0.140 (#3566) * Bump tempfile from 3.16.0 to 3.17.1 (#3539) * Bump tempfile from 3.17.1 to 3.18.0 (#3575) * Bump tempfile from 3.18.0 to 3.19.0 (#3582) * Bump tempfile from 3.19.0 to 3.19.1 (#3588) * Bump thiserror from 2.0.11 to 2.0.12 (#3565) * pre-commit autoupdate (#3547) * pre-commit autoupdate (#3563) * pre-commit autoupdate (#3573) * pre-commit autoupdate (#3580) * pre-commit autoupdate (#3586) * pre-commit autoupdate (#3607) * pre-commit autoupdate (#3615) * pre-commit autoupdate (#3619) * pre-commit autoupdate (#3624) * pre-commit autoupdate (#3633)
This PR implements manifest retrieval from Rust via FFI, and implements the relevant machinery for
RevIndexclasses.Starts from code in #2726, which turns out to be pretty far along!
Tackles #3593