Skip to content

Conversation

dongcarl
Copy link
Contributor

@dongcarl dongcarl commented Jun 7, 2022

This is part of the libbitcoinkernel project: #24303, https://github.com/bitcoin/bitcoin/projects/18


As mentioned in the Stage 1 Step 2 description of the libbitcoinkernel project, ArgsManager will not be part of libbitcoinkernel. Therefore, it is important that we remove any dependence on ArgsManager by code that will be part of libbitcoinkernel. This is the first in a series of PRs aiming to achieve this.

This PR removes CTxMemPool+MempoolAccept's dependency on ArgsManager by introducing a CTxMemPool::Options struct, which is used to specify CTxMemPool's various options at construction time.

These options are:

  • -maxmempool -> CTxMemPool::Options::max_size
  • -mempoolexpiry -> CTxMemPool::Options::expiry
  • -limitancestorcount -> CTxMemPool::Options::limits::ancestor_count
  • -limitancestorsize -> CTxMemPool::Options::limits::ancestor_size
  • -limitdescendantcount -> CTxMemPool::Options::limits::descendant_count
  • -limitdescendantsize -> CTxMemPool::Options::limits::descendant_size

More context can be gleaned from the commit messages. The important commits are:

  • 56eb479 "pool: Add and use MemPoolOptions, ApplyArgsManOptions"
  • a1e08b7 "mempool: Pass in -maxmempool instead of referencing gArgs"
  • 6f4bf3e "mempool: Pass in -mempoolexpiry instead of referencing gArgs"
  • 5958a7f "mempool: Introduce (still-unused) MemPoolLimits"

Reviewers: Help needed in the following commits (see commit messages):

  • a1e08b7 "mempool: Pass in -maxmempool instead of referencing gArgs"
  • 0695081 "node/ifaces: Use existing MemPoolLimits"

Note to Reviewers: There are perhaps an infinite number of ways to architect CTxMemPool::Options, the current one tries to keep it simple, usable, and flexible. I hope we don't spend too much time arguing over the design here since that's not the point. In the case that you're 100% certain that a different design is strictly better than this one in every regard, please show us a fully-implemented branch.


TODO:

  • Use the more ergonomic CTxMemPool::Options where appropriate
  • Doxygen comments for ApplyArgsManOptions, MemPoolOptions

Questions for Reviewers:

  1. Should we use std::chrono::seconds for CTxMemPool::Options::expiry and CTxMemPool::m_expiry instead of an int64_t? Something else? (std::chrono::hours?)
  2. Should I merge CTxMemPool::Limits inside CTxMemPool::Options?

@dongcarl dongcarl force-pushed the 2022-02-libbitcoinkernel-argsman-mempool branch from 42dbf77 to 6abdec9 Compare June 7, 2022 02:07
@DrahtBot
Copy link
Contributor

DrahtBot commented Jun 7, 2022

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #25386 (refactor: Extract MIB_BYTES constant for init.cpp by Empact)
  • #25308 (refactor: Reduce number of LoadChainstate parameters and return values by ryanofsky)
  • #25285 (Add AutoFile without ser-type and ser-version and use it where possible by MarcoFalke)
  • #25193 (indexes: Read the locator's top block during init, allow interaction with reindex-chainstate by mzumsande)
  • #24513 (CChainState -> Chainstate by jamesob)
  • #24158 (Optimize Mempool Reorg logic using Epochs, improving memory usage and runtime. by JeremyRubin)
  • #23443 (p2p: Erlay support signaling by naumenkogs)
  • #13990 (Allow fee estimation to work with lower fees by ajtowns)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@dongcarl
Copy link
Contributor Author

Pushed 532f838 -> 4bbc2e0

  • Added doxygen comments for MemPoolOptions, MemPoolLimits, ApplyArgsManOptions
  • Use CTxMemPool::{Options,Limits} instead of MemPool{Options,Limits} wherever possible
  • Move ApplyArgsManOptions(const ArgsManager&, MemPoolLimits&) to an anonymous namespace in mempool_args.cpp

@dongcarl
Copy link
Contributor Author

Pushed f803134 -> c98d5f1

@dongcarl dongcarl marked this pull request as ready for review June 16, 2022 18:21
@fanquake
Copy link
Member

https://cirrus-ci.com/task/6709025164230656:

Traceback (most recent call last):
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_node.py", line 552, in assert_start_raises_init_error
                                       ret = self.process.wait(timeout=self.rpc_timeout)
                                     File "/usr/lib/python3.6/subprocess.py", line 1469, in wait
                                       raise TimeoutExpired(self.args, timeout)
                                   subprocess.TimeoutExpired: Command '['/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bitcoind', '-datadir=/tmp/cirrus-ci-build/ci/scratch/test_runner/test_runner_\u20bf_🏃_20220616_191835/mempool_limit_214/node0', '-logtimemicros', '-debug', '-debugexclude=libevent', '-debugexclude=leveldb', '-uacomment=testnode0', '-logthreadnames', '-logsourcelocations', '-maxmempool=4']' timed out after 2400 seconds
                                   During handling of the above exception, another exception occurred:
                                   Traceback (most recent call last):
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 133, in main
                                       self.run_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/mempool_limit.py", line 82, in run_test
                                       self.nodes[0].assert_start_raises_init_error(["-maxmempool=4"], "Error: -maxmempool must be at least 5 MB")
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_node.py", line 582, in assert_start_raises_init_error
                                       self._raise_assertion_error(assert_msg)
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_node.py", line 167, in _raise_assertion_error
                                       raise AssertionError(self._node_msg(msg))
                                   AssertionError: [node 0] bitcoind should have exited within 2400s with expected error Error: -maxmempool must be at least 5 MB

Copy link
Member

@glozow glozow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

Re: "Should I merge CTxMemPool::Limits inside CTxMemPool::Options?"

(TLDR imo keeping them separate is fine).
Would it be accurate to think of Limits as policy (used externally in MemPoolAccept) while Options is related to managing the data structure of CTxMemPool itself (i.e. it knows its own check ratio and max size)? Imo separating the ancestor/descendant limits makes sense because they're not just internal to CTxMemPool; they're also used by MemPoolAccept to implement carveouts. For example, it will override them by adding +1 to the descendant limit and setting ancestor limit = 2 for CPFP carve out:

bitcoin/src/validation.cpp

Lines 906 to 908 in 8745296

if (ws.m_vsize > EXTRA_DESCENDANT_TX_SIZE_LIMIT ||
!m_pool.CalculateMemPoolAncestors(*entry, ws.m_ancestors, 2, m_limit_ancestor_size, m_limit_descendants + 1, m_limit_descendant_size + EXTRA_DESCENDANT_TX_SIZE_LIMIT, dummy_err_string)) {
return state.Invalid(TxValidationResult::TX_MEMPOOL_POLICY, "too-long-mempool-chain", errString);

@dongcarl
Copy link
Contributor Author

@MarcoFalke Answered in: #25487 (comment)

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jun 29, 2022
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 17, 2022
The libbitcoin_kernel library should should not be trying to read options from
the command line or make ArgsManager calls. Instead, it should get instead get
options from simple function arguments and options structs.  This commit
removes gArgs accesses from db_wrapper and txdb modules by defining appropriate
options structs, and is a followup to PR's bitcoin#25290 bitcoin#25487 bitcoin#25527 which remove
other ArgsManager uses from kernel modules.

This commit does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 17, 2022
The libbitcoin_kernel library should should not be trying to read options from
the command line or make ArgsManager calls. Instead, it should get instead get
options from simple function arguments and options structs.  This commit
removes gArgs accesses from db_wrapper and txdb modules by defining appropriate
options structs, and is a followup to PR's bitcoin#25290 bitcoin#25487 bitcoin#25527 which remove
other ArgsManager uses from kernel modules.

This commit does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 17, 2022
The libbitcoin_kernel library should should not be trying to read options from
the command line or make ArgsManager calls. Instead, it should get instead get
options from simple function arguments and options structs.  This commit
removes gArgs accesses from dbwrapper and txdb modules by defining appropriate
options structs, and is a followup to PR's bitcoin#25290 bitcoin#25487 bitcoin#25527 which remove
other ArgsManager uses from kernel modules.

This commit does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 17, 2022
The libbitcoin_kernel library should should not be trying to read options from
the command line or make ArgsManager calls. Instead, it should get instead get
options from simple function arguments and options structs.  This commit
removes gArgs accesses from dbwrapper and txdb modules by defining appropriate
options structs, and is a followup to PR's bitcoin#25290 bitcoin#25487 bitcoin#25527 which remove
other ArgsManager uses from kernel modules.

This commit does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 17, 2022
The libbitcoin_kernel library should should not be trying to read options from
the command line or make ArgsManager calls. Instead, it should get instead get
options from simple function arguments and options structs.  This commit
removes gArgs accesses from dbwrapper and txdb modules by defining appropriate
options structs, and is a followup to PR's bitcoin#25290 bitcoin#25487 bitcoin#25527 which remove
other ArgsManager uses from kernel modules.

This commit does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 18, 2022
The libbitcoin_kernel library should should not be trying to read options from
the command line or make ArgsManager calls. Instead, it should get instead get
options from simple function arguments and options structs.  This commit
removes gArgs accesses from dbwrapper and txdb modules by defining appropriate
options structs, and is a followup to PR's bitcoin#25290 bitcoin#25487 bitcoin#25527 which remove
other ArgsManager uses from kernel modules.

This commit does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 22, 2022
The libbitcoin_kernel library should should not be trying to read options from
the command line or make ArgsManager calls. Instead, it should get instead get
options from simple function arguments and options structs.  This commit
removes gArgs accesses from dbwrapper and txdb modules by defining appropriate
options structs, and is a followup to PR's bitcoin#25290 bitcoin#25487 bitcoin#25527 which remove
other ArgsManager uses from kernel modules.

This commit does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 22, 2022
Code in the libbitcoin_kernel library should not be calling ArgsManager methods
or trying to read options from the command line. Instead it should just get
options values from simple structs and function arguments that are passed in
externally. This commit removes gArgs accesses from dbwrapper and txdb modules
by defining appropriate options structs, and is a followup to PR's bitcoin#25290
bitcoin#25487 bitcoin#25527 which remove other ArgsManager calls from kernel modules.

This PR does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 26, 2022
Code in the libbitcoin_kernel library should not be calling ArgsManager methods
or trying to read options from the command line. Instead it should just get
options values from simple structs and function arguments that are passed in
externally. This commit removes gArgs accesses from dbwrapper and txdb modules
by defining appropriate options structs, and is a followup to PR's bitcoin#25290
bitcoin#25487 bitcoin#25527 which remove other ArgsManager calls from kernel modules.

This PR does not change behavior in any way.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Aug 26, 2022
Code in the libbitcoin_kernel library should not be calling ArgsManager methods
or trying to read options from the command line. Instead it should just get
options values from simple structs and function arguments that are passed in
externally. This commit removes gArgs accesses from dbwrapper and txdb modules
by defining appropriate options structs, and is a followup to PR's bitcoin#25290
bitcoin#25487 bitcoin#25527 which remove other ArgsManager calls from kernel modules.

This PR does not change behavior in any way.
glozow added a commit to bitcoin-core/gui that referenced this pull request Oct 9, 2022
33b12e5 docs: improve docs where MemPoolLimits is used (stickies-v)
6945853 test: use NoLimits() in MempoolIndexingTest (stickies-v)
3a86f24 refactor: mempool: use CTxMempool::Limits (stickies-v)
b85af25 refactor: mempool: add MemPoolLimits::NoLimits() (stickies-v)

Pull request description:

  Mempool currently considers 4 limits regarding ancestor and descendant count and size, which get passed around between functions quite a bit. This PR uses `CTxMemPool::Limits` introduced in bitcoin/bitcoin#25290 to simplify those signatures and callsites.

  The purpose of this PR is to improve readability and maintenance, without behaviour change.

  As noted in the first commit "refactor: mempool: change MemPoolLimits members to uint", we currently have an underflow issue where a user could pass a negative `-limitancestorsize`, which is eventually cast to an unsigned integer. This behaviour already exists. Because it's orthogonal and to minimize scope, I think this should be fixed in a separate PR.

ACKs for top commit:
  hebasto:
    ACK 33b12e5, I have reviewed the code and it looks OK, I agree it can be merged.
  glozow:
    reACK 33b12e5

Tree-SHA512: 591c6dcee1894f1c3ca28b34a680eeadcf0d40cda92451b4a422c03087b27d682b5e30ba4367abd75a99b5ccb115b7884b0026958d3c7dddab030549db5a4056
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Oct 14, 2022
Code in the libbitcoin_kernel library should not be calling ArgsManager methods
or trying to read options from the command line. Instead it should just get
options values from simple structs and function arguments that are passed in
externally. This commit removes gArgs accesses from dbwrapper and txdb modules
by defining appropriate options structs, and is a followup to PR's bitcoin#25290
bitcoin#25487 bitcoin#25527 which remove other ArgsManager calls from kernel modules.

This PR does not change behavior in any way.
achow101 added a commit that referenced this pull request Feb 17, 2023
…and txdb

aadd7c5 refactor, validation: Add ChainstateManagerOpts db options (Ryan Ofsky)
0352258 refactor, txdb: Use DBParams struct in CBlockTreeDB (Ryan Ofsky)
c00fa1a refactor, txdb: Add CoinsViewOptions struct (Ryan Ofsky)
2eaeded refactor, dbwrapper: Add DBParams and DBOptions structs (Ryan Ofsky)

Pull request description:

  Code in the libbitcoin_kernel library should not be calling `ArgsManager` methods or trying to read options from the command line. Instead it should just get options values from simple structs and function arguments that are passed in externally. This PR removes `gArgs` accesses from `dbwrapper` and `txdb` modules by defining appropriate options structs, and is a followup to PR's #25290 #25487 #25527 which remove other `ArgsManager` calls from kernel modules.

  This PR does not change behavior in any way. It is a simpler alternative to #25623 because the only thing it does is remove `gArgs` references from kernel code. It avoids other unnecessary changes like adding options to the kernel API (they can be added separately later).

ACKs for top commit:
  TheCharlatan:
    Code review ACK aadd7c5
  achow101:
    ACK aadd7c5
  furszy:
    diff ACK aadd7c5

Tree-SHA512: 46dfd5d99ab3110492e7bba97a87122c831b8344caaf7dd2ebdb6e0ad6aa9174d4d1832d6f3a7465eda9294fe50defaa3c000afbbddc4e72838687df09a63ffd
fanquake added a commit to bitcoin-core/gui that referenced this pull request Mar 14, 2023
…ystem.h

aaced56 refactor: Move error() from util/system.h to logging.h (Ben Woosley)
e7333b4 refactor: Extract util/exception from util/system (Ben Woosley)

Pull request description:

  This pull request is part of the `libbitcoinkernel` project bitcoin/bitcoin#24303 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". These commits were originally authored by empact and are taken from their parent PR #25152.

  #### Context

  There is an ongoing effort to decouple the `ArgsManager` used for command line parsing user-provided arguments from the libbitcoinkernel library (bitcoin/bitcoin#25290, bitcoin/bitcoin#25487, bitcoin/bitcoin#25527, bitcoin/bitcoin#25862, bitcoin/bitcoin#26177, and bitcoin/bitcoin#27125). The `ArgsManager` is defined in `system.h`.

  #### Changes

  Next to providing better code organization, this PR removes some reliance of the tree of libbitcoinkernel header includes on `system.h` (and thus the `ArgsManager` definition) by moving some logging functions out of the `system.*` files.

  Further commits splitting more functionality out of `system.h` are still in #25152 and will be submitted in separate PRs once this PR has been processed.

ACKs for top commit:
  MarcoFalke:
    re-ACK aaced56 🐍

Tree-SHA512: cb39f4cb7a77e7dc1887b1cbf340d53decab8880fc00878a2f12dc627fe67245b4aafd4cc31a9eab0fad1e5bb5d0eb4cdb8d501323ca200fa6ab7b201ae34aea
fanquake added a commit that referenced this pull request Mar 16, 2023
…lity to kernel

b3e78dc refactor: Don't use global chainparams in chainstatemanager method (TheCharlatan)
382b692 Split non/kernel chainparams (Carl Dong)
edabbc7 Add factory functions for Main/Test/Sig/Reg chainparams (Carl Dong)
d938098 Remove UpdateVersionBitsParameters (Carl Dong)
84b8578 Decouple RegTestChainParams from ArgsManager (Carl Dong)
76cd4e7 Decouple SigNetChainParams from ArgsManager (Carl Dong)

Pull request description:

  This pull request is part of the `libbitcoinkernel` project #24303 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". dongcarl is the original author of this patchset, these commits were taken from https://github.com/dongcarl/bitcoin/tree/2022-03-libbitcoinkernel-chainparams-args-only.

  #### Context

  The bitcoin kernel library currently relies on code containing user configurations through the `ArgsManager`. This is not optimal, since as a stand-alone library it should not rely on bitcoind's argument parsing logic. Instead, its interfaces should accept control and options structs that control the kernel library's desired configuration.

  Similar work towards decoupling the `ArgsManager` from the kernel has been done in
  #25290, #25487, #25527 and #25862.

  #### Changes

  By moving the `CChainParams` class definition into the kernel and giving it new factory functions `CChainParams::{RegTest,SigNet,Main,TestNet}`it can be constructed without an `ArgsManager` reference, unlike the current factory function `CreateChainParams`.

  The first few commits remove uses of `ArgsManager` within `CChainParams`. Then the `CChainParams` definition is moved to a new file in the `kernel/` subdirectory.

ACKs for top commit:
  MarcoFalke:
    re-ACK b3e78dc 🛁
  ryanofsky:
    Code review ACK b3e78dc. Only changes since last review were recent review suggestions.
  ajtowns:
    ACK b3e78dc

Tree-SHA512: 3835aca1d3e3c75cc3303dd584bab3a77e58f6c678724a5e359fe4b0e17e0763a00931ee6191f516b9fde50496f59cc691f0709c0254206db3863bbf7ab2cacd
fanquake added a commit to bitcoin-core/gui that referenced this pull request Apr 3, 2023
00e9b97 refactor: Move fs.* to util/fs.* (TheCharlatan)
106b46d Add missing fs.h includes (TheCharlatan)
b202b3d Add missing cstddef include in assumptions.h (TheCharlatan)
18fb363 refactor: Extract util/fs_helpers from util/system (Ben Woosley)

Pull request description:

  This pull request is part of the `libbitcoinkernel` project bitcoin/bitcoin#24303 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel". This commit was originally authored by empact and is taken from its parent PR #25152.

  #### Context

  There is an ongoing effort to decouple the `ArgsManager` used for command line parsing user-provided arguments from the libbitcoinkernel library (bitcoin/bitcoin#25290, bitcoin/bitcoin#25487, bitcoin/bitcoin#25527, bitcoin/bitcoin#25862, bitcoin/bitcoin#26177, and bitcoin/bitcoin#27125). The `ArgsManager` is defined in `system.h`. A similar pull request extracting functionality from `system.h` has been merged in bitcoin/bitcoin#27238.

  #### Changes

  Next to providing better code organization, this PR removes some reliance of the tree of libbitcoinkernel header includes on `system.h` (and thus the `ArgsManager` definition) by moving filesystem related functions out of the `system.*` files.

  There is already a pair of `fs.h` / `fs.cpp` in the top-level `src/` directory. They were not combined with the files introduced here, to keep the patch cleaner and more importantly because they are often included without the utility functions. The new files are therefore named `fs_helpers` and the existing `fs` files are moved into the util directory.

  Further commits splitting more functionality out of `system.h` are still in #25152 and will be submitted in separate PRs once this PR has been processed.

ACKs for top commit:
  hebasto:
    ACK 00e9b97

Tree-SHA512: 31422f148d14ba3c843b99b1550a6fd77c77f350905ca324f93d4f97b652246bc58fa9696c64d1201979cf88733e40be02d262739bb7d417cf22bf506fdb7666
ryanofsky added a commit that referenced this pull request Jun 9, 2023
…base from kernel library

db77f87 scripted-diff: move settings to common namespace (TheCharlatan)
c27e4bd move-only: Move settings to the common library (TheCharlatan)
c2dae5d kernel: Remove chainparams, chainparamsbase, args, settings from kernel library (TheCharlatan)
05870b1 refactor: Remove gArgs access from validation.cpp (TheCharlatan)
8789b11 refactor: Add path argument to FindSnapshotChainstateDir (TheCharlatan)
ef95be3 refactor: Add stop_at_height option in ChainstateManager (TheCharlatan)

Pull request description:

  This pull request is part of the `libbitcoinkernel` project #27587 https://github.com/bitcoin/bitcoin/projects/18 and more specifically its "Step 2: Decouple most non-consensus code from libbitcoinkernel".

  ---

  This completes the removal of the node's chainparams, chainparamsbase, args and settings files and their respective classes from the kernel library. This is the last pull request in a long series working towards decoupling the `ArgsManager` and the `gArgs` global from kernel code. These prior pull requests are: #26177 #27125 #25527 #25487 #25290

ACKs for top commit:
  MarcoFalke:
    lgtm ACK db77f87 🍄
  hebasto:
    ACK db77f87, I have reviewed the code and it looks OK.
  ryanofsky:
    Code review ACK db77f87. Looks great!

Tree-SHA512: cbfbd705d056f2f10f16810d4f869eb152362fff2c5ddae5e1ac6785deae095588e52ad48b29d921962b085e51de1e0ecab6e50f46149ffe3c16250608a2c93a
@bitcoin bitcoin locked and limited conversation to collaborators Jan 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Status: Done or Closed or Rethinking
Development

Successfully merging this pull request may close these issues.