cluster mempool: control/optimize TxGraph memory usage #33157
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/33157. ReviewsSee the guideline for information on the review process. |
This reduces per-Cluster memory usage by making Clusters not aware of their own level. Instead, track it either in calling code, or infer it based on the transactions in them.
ce299a8 to
7547753
Compare
|
🚧 At least one of the CI tasks failed. HintsTry to run the tests locally, according to the documentation. However, a CI failure may still
Leave a comment here, if you need help tracking down a confusing failure. |
Part of #30289.
This adds a few optimizations to reduce
TxGraph's memory usage, and makes sure that dynamic memory it uses doesn't linger after shrinking clusters. Finally, it exposes a functionGetMainMemoryUsage()to computeTxGraph's approximate memory usage.On my 64-bit system, it needs 72 bytes per transaction, 120 bytes per cluster, and 64 bytes per chunk. The code using it also needs a
TxGraph::Refobject per transaction, at 16 bytes per transaction. Overall, this means up to 272 bytes per transaction. Experimenting with this integrated into #28676 shows a total per-transaction overhead of ~720 bytes per mempool transaction, while master has around ~548 bytes per transaction. Or expressed as a ratio of memory usage divided by mempool vsize, 5.8 in master and 6.4 in cluster mempool.When designing
TxGraph, the intent was to make theClustertype a virtual class, with different instances for different cluster sizes (in particular, for singletons) that could optimize memory usage in a more tailored way. I have a very immature idea about getting rid ofDepGraphinstances insideClusterafter #32545, which would cut the increase in overhead down, so I'm not including that here.Another memory usage optimization that may worthwhile, independently of the idea above, is to special-case singleton clusters (given the large fraction of transactions those are) and just store those as a list of transactions rather than having individual
Clusterobjects for them. I'm estimating that could save ~96 bytes per singleton-cluster transaction, but I'm leaving that for a potential future improvement too.