Store frequently accessed header metadata terms in a cache #5619
Conversation
|
To notice the difference between main and PR can also try this trace pattern: MAIN
=>
WITH PRJust |
|
Random thought for a future PR, maybe cache the root nodes from the b-trees as well (with a limit on total size)... |
|
the security object and props are stored in the #db header when we open a db and kept in memory (in couch_db_updater's gen_server state). |
23b3053 to
f4cbd25
Compare
I removed the security_ptr for now as it is cached in the couch_db_updater, but left a note in the comments, that will require a larger change, especially that we read that one from db records across nodes. |
| fsync_options_deprecated, | ||
| cache = #{}, |
There was a problem hiding this comment.
fsync_options_deprecated is a deprecated field and I haven't found any other
source code related to it, so I guess replacing it with cache should be fine.
There was a problem hiding this comment.
yeah, exactly, that one is from when we removed the fsync options (fsync before header, after header, or both, now we always sync both)
Previously, on every write and _all_docs and other call the props term was
always loaded from disk, decompressed and run through binary_to_term. The term
is small, and likely will be in the page cache, but still we'd have to go back
to the disk to fetch it, in the worst case.
To fix it, load the terms once at startup and cache them into memory. This way
we get the best of both worlds: a small header record, since we're using a
pointer to a term, and fast interactive operations, since we don't have to do
extra reads and deserializations.
Technically we could cache the security_ptr term as well, but it is currently
cached in the #db{} record so would require more effort, especially since we
access those across the cluster via fabric_util:get_db() call.
f4cbd25 to
cb46dc1
Compare
Previously, on every write and _all_docs and other call the props term was always loaded from disk, decompressed and run through binary_to_term. The term is small, and likely will be in the page cache, but still we'd have to go back to the disk to fetch it, in the worst case.
To fix it, load the terms once at startup and cache them into memory. This way we get the best of both worlds: a small header record, since we're using a pointer to a term, and fast interactive operations, since we don't have to do extra reads and deserializations.
Technically we could cache the security_ptr term as well, but it is currently cached in the
#db{}record so would require more effort, especially since we access those across the cluster viafabric_util:get_db()call.