cluster mempool: extend DepGraph functionality #30857

sipa · 2024-09-10T00:48:24Z

Part of cluster mempool: #30289

This adds:

DepGraph::AddDependencies to add 0 or more dependencies to a single transaction at once (identical to calling DepGraph::AddDependency once for each, but more efficient).
DepGraph::RemoveTransactions to remove 0 or more transactions from a depgraph.
DepGraph::GetReducedParents (and DepGraph::GetReducedChildren) to get the (reduced) direct parents and children of a transaction in a depgraph.

After which, the Cluster type is removed.

This is the result of fleshing out the design for the "intermediate layer" ("TxGraph", no PR yet) between the cluster linearization layer and the mempool layer. My earlier thinking was that TxGraph would store Cluster objects (vectors of pairs of FeeFracs and sets of parents), and convert them to DepGraph on the fly whenever needed. However, after more consideration, it seems better to have TxGraph store DepGraph objects, and manipulate them directly without constantly re-creating them. This requires DepGraph to have some additional functionality.

The bulk of the complexity here is the addition of DepGraph::RemoveTransactions, which leaves the remaining transactions' positions within the DepGraph untouched (we want existing identifiers to remain valid), so this implies that graphs can now have "holes" (positions that are unused, but followed by positions that are used). To enable that, an extension of the fuzz/test serialization format DepGraphFormatter is included to deal with such holes.

DrahtBot · 2024-09-10T00:48:28Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	instagibbs, ismaelsadeeq, sdaftuar, glozow

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#30605 (Cluster linearization: separate tests from tests-of-tests by sipa)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot · 2024-09-10T01:33:22Z

🚧 At least one of the CI tasks failed.
_{Debug: https://github.com/bitcoin/bitcoin/runs/29906098019}

Hints

Make sure to run all tests locally, according to the documentation.

The failure may happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the
affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

fanquake · 2024-09-10T12:39:36Z

https://cirrus-ci.com/task/4580438562308096?logs=ci#L1723

Run clusterlin_ancestor_finder with args ['/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz', '-runs=1', PosixPath('/ci_container_base/ci/scratch/qa-assets/fuzz_corpora/clusterlin_ancestor_finder')]INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2205311796
INFO: Loaded 1 modules   (615399 inline 8-bit counters): 615399 [0x55b1778f3be8, 0x55b177989fcf), 
INFO: Loaded 1 PC tables (615399 PCs): 615399 [0x55b177989fd0,0x55b1782ede40), 
INFO:       57 files found in /ci_container_base/ci/scratch/qa-assets/fuzz_corpora/clusterlin_ancestor_finder
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 57 min: 1b max: 516b total: 4662b rss: 111Mb
/ci_container_base/src/test/util/cluster_linearize.h:242:29: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'ClusterIndex' (aka 'unsigned int')
    #0 0x55b17470e4e4 in void (anonymous namespace)::DepGraphFormatter::Unser<SpanReader, bitset_detail::IntBitSet<unsigned int>>(SpanReader&, cluster_linearize::DepGraph<bitset_detail::IntBitSet<unsigned int>>&)::'lambda'(unsigned int)::operator()(unsigned int) const ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/./src/test/util/cluster_linearize.h:242:29
    #1 0x55b1746e5efa in void (anonymous namespace)::DepGraphFormatter::Unser<SpanReader, bitset_detail::IntBitSet<unsigned int>>(SpanReader&, cluster_linearize::DepGraph<bitset_detail::IntBitSet<unsigned int>>&) ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/./src/test/util/cluster_linearize.h:268:17
    #2 0x55b1746e5efa in void Wrapper<(anonymous namespace)::DepGraphFormatter, cluster_linearize::DepGraph<bitset_detail::IntBitSet<unsigned int>>&>::Unserialize<SpanReader>(SpanReader&) ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/./src/serialize.h:481:73
    #3 0x55b1746e5efa in void Unserialize<SpanReader, Wrapper<(anonymous namespace)::DepGraphFormatter, cluster_linearize::DepGraph<bitset_detail::IntBitSet<unsigned int>>&>&>(T&, T0&&) ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/./src/serialize.h:761:7
    #4 0x55b1746e5efa in SpanReader& SpanReader::operator>><Wrapper<(anonymous namespace)::DepGraphFormatter, cluster_linearize::DepGraph<bitset_detail::IntBitSet<unsigned int>>&>>(Wrapper<(anonymous namespace)::DepGraphFormatter, cluster_linearize::DepGraph<bitset_detail::IntBitSet<unsigned int>>&>&&) ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/./src/streams.h:114:9
    #5 0x55b1746eb7c4 in clusterlin_ancestor_finder_fuzz_target(std::span<unsigned char const, 18446744073709551615ul>) ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/./src/test/fuzz/cluster_linearize.cpp:454:16
    #6 0x55b174d1203d in std::function<void (std::span<unsigned char const, 18446744073709551615ul>)>::operator()(std::span<unsigned char const, 18446744073709551615ul>) const /usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    #7 0x55b174d1203d in LLVMFuzzerTestOneInput ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/util/./src/test/fuzz/fuzz.cpp:209:5
    #8 0x55b1743ee834 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x1bcb834) (BuildId: c3b4f6b7a5dfbb78ebde90aae8831e87214c3cf3)
    #9 0x55b1743edf29 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x1bcaf29) (BuildId: c3b4f6b7a5dfbb78ebde90aae8831e87214c3cf3)
    #10 0x55b1743efb46 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x1bccb46) (BuildId: c3b4f6b7a5dfbb78ebde90aae8831e87214c3cf3)
    #11 0x55b1743f0057 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x1bcd057) (BuildId: c3b4f6b7a5dfbb78ebde90aae8831e87214c3cf3)
    #12 0x55b1743dd54f in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x1bba54f) (BuildId: c3b4f6b7a5dfbb78ebde90aae8831e87214c3cf3)
    #13 0x55b174407bd6 in main (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x1be4bd6) (BuildId: c3b4f6b7a5dfbb78ebde90aae8831e87214c3cf3)
    #14 0x7fe0a47af1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 6d64b17fbac799e68da7ebd9985ddf9b5cb375e6)
    #15 0x7fe0a47af28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 6d64b17fbac799e68da7ebd9985ddf9b5cb375e6)
    #16 0x55b1743d2534 in _start (/ci_container_base/ci/scratch/build-x86_64-pc-linux-gnu/src/test/fuzz/fuzz+0x1baf534) (BuildId: c3b4f6b7a5dfbb78ebde90aae8831e87214c3cf3)

SUMMARY: UndefinedBehaviorSanitizer: unsigned-integer-overflow /ci_container_base/src/test/util/cluster_linearize.h:242:29 
MS: 0 ; base unit: 0000000000000000000000000000000000000000
0xa,0x7e,0x7e,0x7e,0x3,0x0,0x7e,0x7e,0x7e,0x0,0x8,0x0,0x3f,
\012~~~\003\000~~~\000\010\000?
artifact_prefix='./'; Test unit written to ./crash-8c0d970971a66a2d84b05f68d2956932fb44a077
Base64: Cn5+fgMAfn5+AAgAPw==

The previous releases failure is unrelated.

sipa · 2024-09-10T12:52:46Z

@fanquake Thanks! Fixed, I think.

EDIT: Narrator: it wasn't.

instagibbs · 2024-09-10T16:27:36Z

I'm going to hold off reviewing this until I see the TxGraph PR(dirty draft ok) to better understand its intended usage. 👍

sipa · 2024-09-10T17:14:10Z

@instagibbs Sure.

To give a bit more background, my idea to store things in Cluster form and convert to DepGraph, was due to a (mistaken) belief that incremental changes to a DepGraph would be $\mathcal{O}(n)$ per-dependency, meaning $\mathcal{O}(n^2)$ per-transaction, and thus potentially $\mathcal{O}(n^3)$ for a full $n$ transactions.

@sdaftuar pointed out the possibility of an $\mathcal{O}(n)$ per-transaction algorithm, or $\mathcal{O}(n^2)$ for $n$ transactions, implemented in this PR as DepGraph::AddDependencies. As that's equal in complexity to the current bulk Cluster->DepGraph conversion, there is little reason left to prefer that bulk conversion algorithm, and leaving things in DepGraph form means the same module can be used for more purposes (e.g. computing a transaction's ancestry, as it's literally precomputed already inside DepGraph).

sipa · 2024-09-16T20:44:13Z

Rebased after merge of #30286.

instagibbs

First run through.

suggestion for clusterlin_add_dependencies harness: instagibbs@c5c44da

I think this gets similar or better coverage and is easier for me to understand? (alternatively, take an extra bit and use that as decision bit to batch or individually add, and have the other depgraph do the opposite?)

edit: ah, I'm hodling it wrong! Deleted question.

src/cluster_linearize.h

src/test/util/cluster_linearize.h

src/cluster_linearize.h

src/test/cluster_linearize_tests.cpp

src/test/fuzz/cluster_linearize.cpp

instagibbs · 2024-09-17T16:32:22Z

src/cluster_linearize.h

+    /** Remove the specified positions from this DepGraph.
+     *
+     * The specified positions will no longer be part of Positions(), and dependencies with them are
+     * removed. Note that due to DepGraph only tracking ancestors/descendants (and not direct


Ergonomically speaking, I expect this to be used to remove a set of "direct conflicts" and all descendants of that set, so this wouldn't cause invalid views of the underlying cluster's graph relationships.

Is this correct?

In practice, this will be used to either remove a set of conflicts (together with all their descendants), or to remove a set of mined transactions (together with all their ancestors). In both cases, no transactions will remain which are both ancestors of some deleted transactions and descendants of other deleted transactions.

ah, right! I hadn't considered the mining side. Maybe leave something to that effect?

I feel like that's better left to the layer above; as far as DepGraph is concerned, it's pefectly well-defined to do this, even though we don't care.

src/cluster_linearize.h

instagibbs

it's likely the wrong place for the coverage, but I added some coverage for iterative additional and removal of transactions. Take what you think is useful if any:

diff --git a/src/test/fuzz/cluster_linearize.cpp b/src/test/fuzz/cluster_linearize.cpp
index be06898a6e..62b5ec2343 100644
--- a/src/test/fuzz/cluster_linearize.cpp
+++ b/src/test/fuzz/cluster_linearize.cpp
@@ -1,19 +1,20 @@
 // Copyright (c) The Bitcoin Core developers
 // Distributed under the MIT software license, see the accompanying
 // file COPYING or http://www.opensource.org/licenses/mit-license.php.
 
 #include <cluster_linearize.h>
 #include <serialize.h>
 #include <streams.h>
 #include <test/fuzz/fuzz.h>
 #include <test/fuzz/FuzzedDataProvider.h>
+#include <test/fuzz/util.h>
 #include <test/util/cluster_linearize.h>
 #include <util/bitset.h>
 #include <util/feefrac.h>
 
 #include <algorithm>
 #include <stdint.h>
 #include <vector>
 #include <utility>
 
 using namespace cluster_linearize;
@@ -308,30 +309,89 @@ FUZZ_TARGET(clusterlin_cluster_serialization)
             if (i == j) continue;
             if (provider.ConsumeBool()) cluster[i].second.Set(j);
         }
     }
 
     // Construct dependency graph, and verify it matches the cluster (which includes a round-trip
     // check for the serialization).
     DepGraph depgraph(cluster);
     VerifyDepGraphFromCluster(cluster, depgraph);
 
+    // Copy to store deleted information
+    DepGraph depgraph_copy(depgraph);
+
     // Remove an arbitrary subset (in order to construct a graph with holes) and verify that it
     // still sanity checks (incl. round-tripping serialization).
     uint64_t del = provider.ConsumeIntegralInRange<uint64_t>(1, (uint64_t{1} << num_tx) - 1);
     TestBitSet setdel;
     for (ClusterIndex i : depgraph.Positions()) {
         if (del & 1) setdel.Set(i);
         del >>= 1;
     }
+    assert(del == 0);
+
     depgraph.RemoveTransactions(setdel);
     SanityCheck(depgraph);
+
+    FastRandomContext rng{ConsumeUInt256(provider)};
+
+    // Arbitary addition or removal order for transactions we're deleting
+    std::vector<ClusterIndex> del_indexes;
+    for (const auto del_index : setdel) {
+        del_indexes.push_back(del_index);
+    }
+
+    // Tracks deletion index vs re-addition index for dependency re-addition
+    std::map<ClusterIndex, ClusterIndex> map_del_add_index;
+
+    // Re-add everything back in; should not fail sanity checks
+    // But this will have transactions "out of order" and missing
+    // connections from children to previously deleted ancestors.
+    std::shuffle(del_indexes.begin(), del_indexes.end(), rng);
+    for (ClusterIndex del_index : del_indexes) {
+        const auto add_index{depgraph.AddTransaction(depgraph_copy.FeeRate(del_index))};
+        map_del_add_index[del_index] = add_index;
+    }
+
+    std::shuffle(del_indexes.begin(), del_indexes.end(), rng);
+    for (ClusterIndex del_index : del_indexes) {
+       depgraph.AddDependencies(depgraph_copy.Ancestors(del_index), map_del_add_index[del_index]);
+    }
+
+    SanityCheck(depgraph);
+
+    // But if we repeat this N times, we should get identical depgraphs
+    for (int i = 0; i < provider.ConsumeIntegralInRange<int>(1, 5); ++i) {
+        depgraph_copy = depgraph;
+        // Batch removal or shuffled iterated removal
+        if (provider.ConsumeBool()) {
+            depgraph.RemoveTransactions(setdel);
+        } else {
+            std::shuffle(del_indexes.begin(), del_indexes.end(), rng);
+            for (const auto del_index : del_indexes) {
+                depgraph.RemoveTransactions(TestBitSet::Singleton(del_index));
+            }
+        }
+        SanityCheck(depgraph);
+
+        std::shuffle(del_indexes.begin(), del_indexes.end(), rng);
+        map_del_add_index.clear();
+        for (ClusterIndex del_index : setdel) {
+            const auto add_index{depgraph.AddTransaction(depgraph_copy.FeeRate(del_index))};
+            map_del_add_index[del_index] = add_index;
+        }
+        std::shuffle(del_indexes.begin(), del_indexes.end(), rng);
+        for (ClusterIndex del_index : setdel) {
+            depgraph.AddDependencies(depgraph_copy.Ancestors(del_index), map_del_add_index[del_index]);
+        }
+        assert(depgraph == depgraph_copy);
+    }
 }
 
 FUZZ_TARGET(clusterlin_depgraph_serialization)
 {
     // Verify that any deserialized depgraph is acyclic and roundtrips to an identical depgraph.
 
     // Construct a graph by deserializing.
     SpanReader reader(buffer);
     DepGraph<TestBitSet> depgraph;
     try {

src/cluster_linearize.h

sipa · 2024-09-18T00:32:02Z

@glozow It's already used in two places in this PR.

Maybe I should clarify in the PR description that this is just a generic utility feature, not something the next layer up will need per se (AFAIK). But because I was introducing a second place where it's invoked, and it's already used somewhere in the test, I felt it made sense to turn it into a proper DepGraph function, with proper tests.

glozow · 2024-09-18T00:44:37Z

Sorry, I don't know how I missed the calls within this PR 🤦 I think I was grepping only on the first commit.

sipa

I addressed most of the feedback received. Thanks @instagibbs and @glozow!

In addition, the DepGraphFormatter::Unser logic was changed significantly. The result is simpler to follow I think (doesn't involve adding a transaction once and then undoing the insertion and redoing it once the position is known), and reduces the complexity from $\mathcal{O}(n^3)$ to $\mathcal{O}(n^2)$ in the process. For this, the commits were reordered, and their commit messages expanded to explain the rationale.

@instagibbs I have added your testing of re-adding in clusterlin_cluster_serialization as a separate commit with some minor changes (mostly to keep the result efficient).

src/cluster_linearize.h

sipa · 2024-09-18T17:47:23Z

src/cluster_linearize.h

+    /** Remove the specified positions from this DepGraph.
+     *
+     * The specified positions will no longer be part of Positions(), and dependencies with them are
+     * removed. Note that due to DepGraph only tracking ancestors/descendants (and not direct


I feel like that's better left to the layer above; as far as DepGraph is concerned, it's pefectly well-defined to do this, even though we don't care.

src/cluster_linearize.h

src/test/fuzz/cluster_linearize.cpp

src/test/cluster_linearize_tests.cpp

src/test/util/cluster_linearize.h

src/cluster_linearize.h

instagibbs · 2024-09-18T19:01:42Z

looked at git range-diff master 9f67d6dcdcf7e9c75a93ad78f8902537599f4f6d 4476915bc76b48eff9b5e67fd6bd7647cc12793f

LGTM, though I haven't re-validated the serializer, trusting the tests of tests. I want to think more about the fuzz targets to see if there are any other easy pickups in coverage.

sipa · 2024-09-30T22:33:48Z

Sorry for the repeated force pushes; I made a change in the wrong commit, and then made some minor improvements to the new fuzz too.

Absent non-nitty review that needs addressing, I don't intend to make change here anymore.

instagibbs · 2024-10-01T00:48:01Z

reACK 2b21aeb

via git range-diff master 7051fcdda6fdc95677a34d5c14321d7580eceed8 2b21aebd9754c503bac12d1dbf566b3aa27451e8

sdaftuar

Code review ACK, just some nits.

src/test/util/cluster_linearize.h

src/cluster_linearize.h

src/test/util/cluster_linearize.h

sdaftuar · 2024-10-01T23:42:31Z

ACK 2b21aeb

ismaelsadeeq

Concept ACK

ismaelsadeeq

first pass light code review ACK 2b21aeb

Mostly trying to understand what the code is doing!

comments are just nits

src/test/util/cluster_linearize.h

src/cluster_linearize.h

A fuzz test already relies on these operations, and a future commit will need the same logic too. Therefore, abstract them out into proper member functions, with proper testing.

This commit does not change the serialization format. Its purpose is making a few changes already in order to reduce the diff size of the later commit that introduces support for holes in DepGraph. The previous approach was to immediately construct a transaction as soon as its feerate was known in a preliminary position, and then undo that, and place it in the correct position once the position information is known (such that a deserialization error in between would not result in an inconsistent state). The new approach is to delay the actual transaction creation until all its information is known, avoiding the need to undo and redo. This requires a different means of determining whether dependencies are redundant, but that has the advantage that a later commit can apply all dependencies at once, reducing the complexity of deserialization.

This does not change the serialization format. It turns out that it is unnecessary to keep track of the order of transactions in the so-far reconstructed DepGraph to decide how far from the end to insert a new transaction.

…s at once This changes DepGraph::AddDependency into DepGraph::AddDependencies, which takes in a single child, but a set of parent transactions, making them all dependencies at once. This is important for performance. N transactions can have O(N^2) parents combined, so constructing a full DepGraph using just AddDependency (which is O(N) on its own) could take O(N^3) time, while doing the same with AddDependencies (also O(N) on its own) only takes O(N^2). Notably, this matters for DepGraphFormatter::Unser, which goes from O(N^3) to O(N^2). Co-Authored-By: Greg Sanders <gsanders87@gmail.com>

… DepGraph This commits introduces support in DepGraph for the transaction positions to be non-continuous. Specifically, it adds: * DepGraph::RemoveTransactions which removes 0 or more positions from a DepGraph. * DepGraph::Positions() to get a set of which positions are in use. * DepGraph::PositionRange() to get the highest used position in a DepGraph + 1. In addition, it extends the DepGraphFormatter format to support holes in a compatible way (it serializes non-holey DepGraphs identically to the old code, and deserializes them the same way)

This combines the clusterlin_add_dependency and clusterlin_cluster_serialization fuzz tests into a single clusterlin_depgraph_sim fuzz test. This tests starts from an empty DepGraph and performs a arbitrary number of AddTransaction, AddDependencies, and RemoveTransactions operations on it, and compares the resulting state with a naive reimplementation.

instagibbs

reACK 0b3ec8c

via git range-diff master 2b21aebd9754c503bac12d1dbf566b3aa27451e8 0b3ec8c59b2b6d598531cd4e688eb276e597c825

ismaelsadeeq

reACK 0b3ec8c

sdaftuar · 2024-10-08T22:42:10Z

reACK 0b3ec8c

glozow

ACK 0b3ec8c, reviewed range-diff from aab53dd and clusterlin_depgraph_sim

glozow · 2024-10-10T14:10:14Z

src/cluster_linearize.h

    {
-        // Determine reordering mapping, by sorting by decreasing feerate.
-        std::iota(m_sorted_to_original.begin(), m_sorted_to_original.end(), ClusterIndex{0});
+        // Determine reordering mapping, by sorting by decreasing feerate. Unusued positions are


tiny typo unusued

glozow · 2024-10-10T14:21:24Z

src/cluster_linearize.h

@@ -19,14 +19,6 @@

 namespace cluster_linearize {

-/** Data type to represent cluster input.


Nice, no more naming clash for Cluster in my mind

…47757ea3b 1047757ea3b kernel: Add pure kernel bitcoin-chainstate c568fdf75fd kernel: Add block index utility functions to C header 0f1da1dcba5 kernel: Add function to read block undo data from disk to C header 45af559c9f6 kernel: Add functions to read block from disk to C header 2a7f8a8240c kernel: Add function for copying block data to C header b19f5336c03 kernel: Add functions for the block validation state to C header 9c0ffa913f4 kernel: Add validation interface to C header a93318c6152 kernel: Add interrupt function to C header 51053f33720 kernel: Add import blocks function to C header 6b0ada2af42 kernel: Add chainstate load options for in-memory dbs in C header 34427bfa9c7 kernel: Add options for reindexing in C header ca57311c969 kernel: Add block validation to C header 44156d84838 Kernel: Add chainstate loading to kernel C header 2cee46cdcc1 kernel: Add chainstate manager object to C header 7102c7ae45e kernel: Add notifications context option to C header ed628a2a3c4 kerenl: Add chain params context option to C header 27643297ff7 kernel: Add kernel library context object 2ba22cf3f90 kernel: Add logging to kernel library C header 873874c03e9 kernel: Introduce initial kernel C header API d94adc7270b Merge bitcoin/bitcoin#29702: fees: Remove CLIENT_VERSION serialization 7290bc61c00 Merge bitcoin/bitcoin#31078: build: Fix kernel static lib component install 68f29b24907 Merge bitcoin/bitcoin#31141: doc: Make list of targets in depends README consistent e9b95665eea Merge bitcoin/bitcoin#31046: init: Some small chainstate load improvements b8c821cc1ea Merge bitcoin/bitcoin#30724: test: add test for specifying custom pidfile via `-pid` a0c9595810c doc: Make list of targets in depends README consistent fa1c5cc9df1 fees: Log non-fatal errors as [warning], instead of info-level ffe4261cb06 Merge bitcoin/bitcoin#30935: ci: Approximate MAKEJOBS in image build phase 28ce159bc32 Merge bitcoin/bitcoin#30183: rpc: net: follow-ups for #30062 684873931b3 Merge bitcoin/bitcoin#26334: Add Signet and testnet4 launch shortcuts for Windows 9b0e2598089 Merge bitcoin/bitcoin#31121: guix: Enable CET for `glibc` package d9f8dc64534 Merge bitcoin/bitcoin#31097: validation: Improve input script check error reporting a16917fb598 rpc, net: improve `mapped_as` doc for getrawaddrman/getpeerinfo 563c4d29268 Merge bitcoin/bitcoin#31105: Update libmultiprocess library 0e9f20625a1 Merge bitcoin/bitcoin#31063: lint: commit-script-check.sh: echo to stderr e8f72aefd20 Merge bitcoin/bitcoin#29877: tracing: explicitly cast block_connected duration to nanoseconds 86e2a6b749c [test] A non-standard transaction which is also consensus-invalid should return the consensus error 4d3da08d1b9 guix: Enable CET for `glibc` package a38603456e9 Merge bitcoin/bitcoin#31100: doc: remove dependency install instructions from win docs 90b405516f7 Update libmultiprocess library 479715e9db0 Merge bitcoin/bitcoin#30996: doc: update signet documentation related to build directories 99e041f86fd Merge bitcoin/bitcoin#31099: doc: drop macOS LLVM install instructions 21e2f06a1cc Merge bitcoin/bitcoin#31067: test: Print CompletedProcess object on error 184f12c1542 doc: remove dependency install instructions from win docs dea9fb9a8b8 Merge bitcoin/bitcoin#30093: optimization: reserve memory allocation for transaction inputs/outputs 79aa8280b2e doc: drop LLVM install instructions 2123c94448e Merge bitcoin/bitcoin#30527: Bump python minimum supported version to 3.10 538ccaed004 Merge bitcoin/bitcoin#31048: build: Bump minimum supported macOS to 13.0 f859ff8a4e9 [validation] Improve script check error reporting ddddbac9c10 fees: Pin required version to 149900 fa5126adcb1 fees: Pin "version that wrote" to 0 0ca1d1bf69c Merge bitcoin/bitcoin#31092: doc: fuzz: remove Honggfuzz NetDriver instructions d823ba6e20b doc: fuzz: remove Honggfuzz NetDriver instructions 15563d3388e Merge bitcoin/bitcoin#30859: doc: cmake: prepend "build" to functional/test_runner.py 2ac5ba24bf0 Merge bitcoin/bitcoin#31083: doc: add doxygen for m_args in tests a0e089a71dc build: Bump minimum supported macOS to 13.0 1fe1b3ba8e9 doc: doxygen comment for m_args usage in tests 82e16e69832 cmake: Refactor install kernel dependencies 42e62779873 build: Add static libraries to Kernel install component e64b2f1a16e doc: cmake: prepend and explain "build/" where needed 48cf3da6360 Merge bitcoin/bitcoin#30970: build: Add missing USDT header dependency to kernel d8b835cf18c Merge bitcoin/bitcoin#31070: contrib: fix typos in check-deps.sh da8824ba301 Fix typos in check-deps.sh fa43c4f93ca test: Print CompletedProcess object on error 489e5aa3a29 Merge bitcoin/bitcoin#30857: cluster mempool: extend DepGraph functionality 9f45062b9b0 Merge bitcoin/bitcoin#30937: build: scripted-diff: drop config/ subdir for bitcoin-config.h 882f736d0a6 doc: lint: correct outdated comment (s/Makefile.am/CMakeLists.txt/) 1786be7b4a5 scripted-diff: drop config/ subdir for bitcoin-config.h, rename to bitcoin-build-config.h 0c2c3bb3f5c Merge bitcoin/bitcoin#30955: Mining interface: getCoinbaseMerklePath() and submitSolution() 9909a34d794 Merge bitcoin/bitcoin#30992: doc: update IBD requirements in doc/README.md fac6cfe5ac0 lint: commit-script-check.sh: echo to stderr 5fb94550638 Merge bitcoin/bitcoin#31058: refactor: include the proper header rather than forward-declaring RemovalReasonToString e569eb8d917 Merge bitcoin/bitcoin#30885: scripted-diff: Modernize nLocalServices naming 31cc5006c3d init: Return fatal failure on snapshot validation failure 8f1246e8338 init: Improve chainstate init db error messages 5837e3463fe Merge bitcoin/bitcoin#30967: refactor: Replace g_genesis_wait_cv with m_tip_block_cv ca2e4ba352c refactor: include the proper header rather than forward-declaring RemovalReasonToString a9f6a57b691 Merge bitcoin/bitcoin#30920: test: Remove 0.16.3 test from wallet_backwards_compatibility.py fa71bedf860 ci: Approximate MAKEJOBS in image build phase 03696bb1bd5 Merge bitcoin/bitcoin#31045: ci: Add missing -DWERROR=ON to test-each-commit 56093565bbe Merge bitcoin/bitcoin#31018: test: Treat exclude list warning as failure in CI bb47b5a6576 Merge bitcoin/bitcoin#31038: test: Fix copy-paste in wallet/test/db_tests ostream operator 3fecf36c7b3 Merge bitcoin/bitcoin#31056: ci: Double ctest timeout 3c4a9419dbe Merge bitcoin/bitcoin#31013: depends: For mingw cross compile use -gcc-posix to prevent library conflict 5d5cc021ce3 Merge bitcoin/bitcoin#31051: test: remove unused code from `script_tests` caf44e500eb Merge bitcoin/bitcoin#31008: depends: Print ready-to-use `--toolchain` option for CMake invocation fa5ebc99207 ci: Double ctest timeout 0b3ec8c59b2 clusterlin: remove Cluster type 1c24c625105 clusterlin: merge two DepGraph fuzz tests into simulation test 0606e66fdbb clusterlin: add DepGraph::RemoveTransactions and support for holes in DepGraph 75b5d42419e clusterlin: make DepGraph::AddDependency support multiple dependencies at once abf50649d13 clusterlin: simplify DepGraphFormatter::Ser eaab55ffc81 clusterlin: rework DepGraphFormatter::Unser 5901cf7100a clusterlin: abstract out DepGraph::GetReduced{Parents,Children} e0287bc4b2d test: remove unused code from script_tests 62e45167221 Merge bitcoin/bitcoin#31026: ci: set a ctest test timeout of 1200 (20 minutes) cd093049dda init: Remove incorrect comment about shutdown condition 635e9f85d76 init: Remove misleading log line when user chooses not to retry fa1cffacae6 ci: Install missing nproc in macos task faf7a2bccc7 ci: Add missing -DWERROR=ON to test-each-commit 56aad83307e ci: set a ctest timeout of 1200 (20 minutes) 1b707146717 Merge bitcoin-core/gui#840: qt6: Handle different signatures of `QANEF::nativeEventFilter` 720ce880a35 init: Improve comment describing chainstate load retry behaviour baea842ff18 init: Remove unneeded argument for mempool_opts checks ec58dfe8f74 Merge bitcoin/bitcoin#31010: cmake: Avoid hardcoding Qt's major version in Find module / variable names 5fe6878b5f7 Merge bitcoin-core/gui#836: Fix display issues for IPv6 proxy setup in Options Dialog (UI only, no functionality impact) f50557f5d36 test: Fix copy-paste in db_tests ostream operator 5ea335a97f8 Merge bitcoin/bitcoin#30793: rpc: add getorphantxs 76e2e8aabd8 Merge bitcoin/bitcoin#31035: doc: Archive 28.0 release notes f019fcec412 doc: Archive 28.0 release notes 80761afced1 qt6: Handle different signatures of `QANEF::nativeEventFilter` 51c698161b5 Merge bitcoin-core/gui#837: qt6: Fix linking when configured with `-DENABLE_WALLET=OFF` 4be785b3e33 Merge bitcoin-core/gui#839: qt6, test: Handle deprecated code f117f3f7473 Merge bitcoin-core/gui#838: qt6: Handle deprecated `QLocale::nativeCountryName` 5625840c11d qt6, test: Handle deprecated `QVERIFY_EXCEPTION_THROWN` 772928a13c2 Merge bitcoin/bitcoin#30982: docs: Add instructions on how to self-sign bitcoin-core binaries for macOS 27709f51ee0 docs: Add instructions on how to self-sign bitcoin-core binaries for macOS cfb59da4b3b Merge bitcoin/bitcoin#30980: fuzz: fix bug in p2p_headers_presync harness dda2613239b Merge bitcoin/bitcoin#30929: log: Enforce trailing newline e0ae9c14c4e Merge bitcoin/bitcoin#31011: refactor: move util/pcp and util/netif to common/ 98c1536852d test: add getorphantxs tests 93f48fceb7d test: add tx_in_orphanage() 34a9c10e8cd rpc: add getorphantxs f511ff3654d refactor: move verbosity parsing to rpc/util 36a6d4b0078 doc: update IBD requirements in doc/README.md fa6d14eacb2 test: Treat exclude list warning as failure in CI 6a370435526 Merge bitcoin/bitcoin#31007: doc: add testnet4 section header for config file 70910eb2ecb Merge bitcoin/bitcoin#31016: test: add missing sync to feature_fee_estimation.py 532491faf1a net: add GetOrphanTransactions() to PeerManager 91b65adff2a refactor: add OrphanTxBase for external use a1576edab35 test: add missing sync to feature_fee_estimation.py ae56b3230b2 depends: For mingw cross compile use -gcc-posix to prevent library conflict fd38711217c ci: make CI job fail when check-deps.sh script fails d51edecddcb common: move pcp.cpp and netif.cpp files from util to common library since they depend on netaddress.cpp 61cdb1c9d83 doc: add testnet4 section header for config file deacf3c7cd6 cmake: Avoid hardcoding Qt's major version in Find module 605926da0ab depends: Print ready-to-use `--toolchain` option for CMake invocation fa2b7d8d6b3 Remove redundant unterminated-logprintf tidy check bbbb2e43ee9 log: Enforce trailing newline, Remove redundant m_started_new_line fa22e5c430a refactor: Remove dead code that assumed tip == nullptr fa2e4439652 refactor: Replace g_genesis_wait_cv with m_tip_block_cv fa7f52af1a4 refactor: Use wait_for predicate to check for interrupt 5ca28ef28bc refactor: Split up NodeContext shutdown_signal and shutdown_request fad8e7fba7b bugfix: Mark m_tip_block_cv as guarded by m_tip_block_mutex fa18586c29d refactor: Add missing GUARDED_BY(m_tip_block_mutex) fa4c0750331 doc: Clarify waitTipChanged docs fc642c33ef2 Merge bitcoin/bitcoin#30718: test: switch MiniWallet padding unit from weight to vsize d7f956a309e Merge bitcoin/bitcoin#30968: init: Remove retry for loop c33eb2360e2 Merge bitcoin/bitcoin#30043: net: Replace libnatpmp with built-in PCP+NATPMP implementation f3c74c4a7e1 Merge bitcoin/bitcoin#30989: guix: Drop no longer needed `PATH` modification 5c7cacf649a ci: Remove natpmp build option and libnatpmp dependency 7e7ec984da5 doc: Remove mention of natpmp build options 061c3e32a26 depends: Drop natpmp and associated option from depends 20a18bf6aa3 build: Drop libnatpmp from build system 7b04709862f qt: Changes for built-in PCP+NAT-PMP 52f8ef66c61 net: Replace libnatpmp with built-in NATPMP+PCP implementation in mapport 97c97177cdb net: Add PCP and NATPMP implementation cb750b4b405 qt6, test: Use `qWarning()` instead of `QWARN()` macro 9123a286e97 qt6: Handle deprecated `QLocale::nativeCountryName` 940edd6ac24 test: refactor: introduce and use `TRUC_CHILD_MAX_VSIZE` constant c16ae717689 test: switch MiniWallet padding unit from weight to vsize a647d4400d5 doc: update signet documentation related to build directories f1daa80521e guix: Drop no longer needed `PATH` modification d812cf11896 Merge bitcoin/bitcoin#30879: test: re-bucket long-running tests 18d4c43cab4 Merge bitcoin/bitcoin#30921: test: generalize HasReason and use it in FailFmtWithError d7fcc91416a Merge bitcoin/bitcoin#30974: ci: Inline PACKAGE_MANAGER_INSTALL 29d00a1cee1 Merge bitcoin/bitcoin#30940: depends: Fix build with `MULTIPROCESS=1` in Guix environment 89a8e9b732f Merge bitcoin/bitcoin#30979: contrib: Update asmap link in seeds readme fafd1a0f648 ci: Inline PACKAGE_MANAGER_INSTALL 36ad9516dbd Merge bitcoin/bitcoin#30981: ci: add timestamps to cirrus jobs fa7c2838a5f Merge bitcoin/bitcoin#30948: test: Add missing sync_mempools() to fill_mempool() f951f1fab25 ci: add timestamps to cirrus jobs f158993fd55 contrib: Update asmap link in seeds readme d5af7d28f47 Merge bitcoin/bitcoin#30976: depends, doc: Drop package-specific note about CMake a7498cc7e26 Fix bug in p2p_headers_presync harness 4cf84b344de depends, doc: No need to specify general requirement e13da501db9 Merge bitcoin/bitcoin#30973: doc: fix `loadtxoutset` example 513b7136c79 Merge bitcoin/bitcoin#30961: ci: add `LLVM_SYMBOLIZER_PATH` to Valgrind fuzz job 286725168ae doc: fix loadtxoutset example 525e9dcba0b Add submitSolution to BlockTemplate interface 47b4875ef05 Add getCoinbaseMerklePath() to Mining interface 63d6ad7c89c Move BlockMerkleBranch back to merkle.{h,cpp} 65f6e7078b1 Merge bitcoin/bitcoin#30510: multiprocess: Add IPC wrapper for Mining interface da612cea032 Merge bitcoin/bitcoin#30962: validation: Disable CheckForkWarningConditions for background chainstate e9d60af9889 refactor: Replace init retry for loop with if statement c1d8870ea41 refactor: Move most of init retry for loop to a function ccd10fdb97f build: Add missing USDT header dependency to kernel 781c01f5806 init: Check mempool arguments in AppInitParameterInteractions 39219fe145e Merge bitcoin/bitcoin#30946: doc: correct the zmq automatic build info 06e7e836329 doc: correct the zmq automatic build info a9773b6215e Merge bitcoin/bitcoin#30963: doc: Adjust links in OSS-Fuzz section fa6c1946d23 doc: Adjust links in OSS-Fuzz section c0a0c72b4d6 validation: Disable CheckForkWarningConditions for background chainstate c1832584bfd ci: add LLVM_SYMBOLIZER_PATH to Valgrind fuzz job 393f323bd60 Merge bitcoin/bitcoin#30952: test: Use shell builtins in run_command test case faf801515f8 test: Add missing sync_mempools() to fill_mempool() fa48be6f023 test: Refactor fill_mempool to extract send_batch helper 1a332817665 doc: multiprocess documentation improvements 90a5786bba4 Merge bitcoin/bitcoin#30678: wallet: Write best block to disk before backup d043950ba24 multiprocess: Add serialization code for BlockValidationState 33c2eee285e multiprocess: Add IPC wrapper for Mining interface 06882f84017 multiprocess: Add serialization code for vector<char> 095286f790a multiprocess: Add serialization code for CTransaction 69dfeb18761 multiprocess: update common-types.h to use C++20 concepts 206c6e78eec build: Make bitcoin_ipc_test depend on bitcoin_ipc 070e6a32d5f depends: Update libmultiprocess library for cmake headers target dabc74e86c3 Merge bitcoin/bitcoin#30409: Introduce waitTipChanged() mining interface, replace RPCNotifyBlockChange, drop CRPCSignals & g_best_block 7bd3ee62f6d test: Use shell builtins in run_command test case f5a2000579b test: re-bucket long-running tests 06b4c339e89 depends: Fix reproducibility when building with `MULTIPROCESS=1` 04e4d52420a test: add test for specifying custom pidfile via `-pid` b832ffe0446 refactor: introduce default pid file name constant in tests d8e3afc3352 depends: Fix build with `MULTIPROCESS=1` in Guix environment f20fe33e94c test: Add basic balance coverage to wallet_assumeutxo.py d72df63d169 net: Use GetLocalAddresses in Discover e02030432b7 net: Add netif utility 754e4254388 crypto: Add missing WriteBE16 function 33adc7521cc Merge bitcoin/bitcoin#30765: refactor: Allow `CScript`'s `operator<<` to accept spans, not just vectors 0894748316c Merge bitcoin/bitcoin#30918: fuzz: Add check in `p2p_headers_presync` that chain work never exceeds minimum work f57a6754ed6 Merge bitcoin/bitcoin#30826: fuzz: reduce number of iterations in `crypto_aeadchacha20poly1305` target 48c20dbd86c Merge bitcoin/bitcoin#30794: interpreter: use int32_t instead of int type for risczero compile 4148e60909e Merge bitcoin/bitcoin#30679: fix: handle invalid `-rpcbind` port earlier a8a2628b7a9 Merge bitcoin/bitcoin#30828: interfaces: #30697 follow ups 0d81b3ddedc Merge bitcoin/bitcoin#30568: addrman: change internal id counting to int64_t c985a34b9c3 Merge bitcoin/bitcoin#26990: cli: Improve error message on multiwallet cli-side commands 037b101e808 test: Add coverage for best block locator write in wallet_backup 31c0df03890 wallet: migration, write best locator before unloading wallet 7e3dbe4180c wallet: Write best block to disk before backup 79f20fa1b1e Merge bitcoin/bitcoin#30561: refactor: move `SignSignature` helpers to test utils 284bd17309a add check that chainwork doesn't exceed minimum work 9aa5d1c3fcd add clarification in comment 197aa249551 Merge bitcoin/bitcoin#30856: build: drop obj/ subdirectory for generated build.h 7025942687f build: drop superfluous `HAVE_BUILD_INFO` define 0dd662510c5 build: drop obj/ subdir for generated build.h, rename to bitcoin-build-info.h 84cd6478c42 Merge bitcoin/bitcoin#30927: Follow-up after AutoFile position caching: remove unused code caac06f784c streams: reorder/document functions 67a3d590768 streams: remove unused code 2db926f49c8 Merge bitcoin/bitcoin#30889: log: Use ConstevalFormatString fee4cba4847 gui: Fix proxy details display in Options Dialog 9ba56884f62 Merge bitcoin/bitcoin#30869: ci: Print inner env, Make ccache config more flexible 6c3c619b35c test: generalize HasReason and use it in FailFmtWithError ab0b5706b25 Merge bitcoin/bitcoin#30875: doc: fixed inconsistencies in documentation between autotools to cmake change fd08fded63a Merge bitcoin/bitcoin#30639: ci: Use clang-19 in msan tasks 5be34bacf6d qt: Fix linking when configured with `-DENABLE_WALLET=OFF` a9964c04447 doc: Updating docs from autotools to cmake fae44c83da9 test: Remove 0.16.3 test from wallet_backwards_compatibility.py 69409bc6e55 Merge bitcoin/bitcoin#30908: doc: remove Eclipser fuzzing documentation 6b97882ab53 Merge bitcoin/bitcoin#30915: ci: Use `ninja` to build in macOS native CI job e6994efe08b fix: increase rpcbind check robustness d38e3aed89e fix: handle invalid rpcbind port earlier 83b67f2e6d5 refactor: move host/port checking 73c243965ab test: add tests for invalid rpcbind ports 54227e681a4 rpc, cli: improve error message on multiwallet mode 735436df8ce Remove outdated Eclipser fuzzing documentation ccccb67851b ci: Use clang-19 in msan tasks facbcd4cef8 log: Use ConstevalFormatString d01b85bfecb ci: Use `ninja` to build in macOS native CI job 6fc46927971 Merge bitcoin/bitcoin#29624: doc: update NeedsRedownload() and nStatus comment 2a0949f0977 Merge bitcoin/bitcoin#30888: build: optimize .h generation in GenerateHeaderFrom{Raw,Json}.cmake bdbc90f29ac Merge bitcoin/bitcoin#30902: Remove Autotools packages from CI (and depends doc) a95e742b692 Merge bitcoin/bitcoin#30913: ci: Use macos-14 GHA image (x86_64-apple-darwin22.6.0 -> arm64-apple-darwin23.6.0) 225718eda89 Merge bitcoin/bitcoin#30438: guix: (explicitly) build Linux GCC with `--enable-cet` fab932b4211 ci: Remove incorrectly hardcoded HOST in mac_native task af9f9878934 doc: update NeedsRedownload() comment fa8f35d7865 ci: Use macos-14 GHA image 7942951e3fc Remove unused g_best_block e3a560ca68d rpc: use waitTipChanged for longpoll 460687a09c2 Remove unused CRPCSignals dca923150e3 Replace RPCNotifyBlockChange with waitTipChanged() 2a40ee11219 rpc: check for negative timeout arg in waitfor* de7c855b3af rpc: recommend -rpcclienttimeout=0 for waitfor* 77ec072925a rpc: fix waitfornewblock description 285fe9fb51c rpc: add test for waitforblock and waitfornewblock b94b27cf05c Add waitTipChanged to Mining interface 7eccdaf1608 node: Track last block that received a blockTip notification ebb8215f236 Rename getTipHash() to getTip() and return BlockRef 89a8f74bbbb refactor: rename BlockKey to BlockRef 9f1aa88d4d9 Merge bitcoin/bitcoin#30884: streams: cache file position within AutoFile 06329eb1348 Merge bitcoin/bitcoin#29436: net: call `Select` with reachable networks in `ThreadOpenConnections` e983ed41d9f Merge bitcoin/bitcoin#30410: rpc, rest: Improve block rpc error handling, check header before attempting to read block data. fce9e065c16 Merge bitcoin/bitcoin#28358: Drop -dbcache limit 8d000b85dd4 Merge bitcoin/bitcoin#30868: refactor: add clang-tidy `modernize-use-starts-ends-with` check 3f66642820b Merge bitcoin/bitcoin#30440: Have createNewBlock() return a BlockTemplate interface 2bf721e76a5 Merge bitcoin/bitcoin#30661: fuzz: Test headers pre-sync through p2p c38e9993de7 Merge bitcoin/bitcoin#30286: cluster mempool: optimized candidate search fa99e4521b6 ci: Allow CCACHE_DIR bind mount 37679b856ce Merge bitcoin/bitcoin#30899: qt: Translations update fc7b507e9a5 tidy: add clang-tidy `modernize-use-starts-ends-with` check 7a8a6a06676 doc: Fix comment in `contrib/devtools/check-deps.sh` script 712d105e093 depends, doc: Do not install Autotools packages b786449e663 ci: Do not install Autotools packages a240e150e83 streams: remove AutoFile::Get() entirely ae052957614 qt: Translations update 6a1aa510e31 rpc: check block index before reading block / undo data 6cbf2e5f819 rpc: Improve gettxoutproof error when only header is available. 69fc867ea19 test: add coverage to getblock and getblockstats 5290cbd5850 rpc: Improve getblock / getblockstats error when only header is available. e5b537bbdfb rest: improve error when only header of a block is available. e624a9bef16 streams: cache file position within AutoFile 89bf11b8072 guix: build Linux GCC with --enable-cet a93c171faa7 Drop unneeded nullptr check from CreateNewBlock() dd87b6dff35 Have createNewBlock return BlockTemplate interface fae9b60c4ff test: Use LogPrintStr to test m_log_sourcelocations 33381ea530a scripted-diff: Modernize nLocalServices to m_local_services 2a581144f28 build: Minimize I/O operations in GenerateHeaderFromJson.cmake aa003d1568b build: Minimize I/O operations in GenerateHeaderFromRaw.cmake 9ad2fe7e69e clusterlin: only start/use search when enough iterations left bd044356edb clusterlin: improve heuristic to decide split transaction (optimization) 71f26293988 clusterlin: include topological pot subsets automatically (optimization) e20fda77a2d clusterlin: reduce computation of unnecessary pot sets (optimization) 6060a948caf clusterlin bench: add example hard cluster benchmarks 2965fbf203f clusterlin: track upper bound potential set for work items (optimization) 9e43e4ce109 clusterlin: use feerate-sorted depgraph in SearchCandidateFinder b80e6dfe780 clusterlin: add reordering support for DepGraph 85a285a3061 clusterlin: separate initial search entries per component (optimization) e4faea9ca79 clusterlin bench: have low/high iter benchmarks instead of per-iter fa39b1ca638 doc: move-only logging warning fa252da0b9c ci: Remove hardcoded CCACHE_DIR in cirrus fa146904e19 ci: Bump default CCACHE_MAXSIZE to 500M aaaa7cf8bad cirrus: Drop CCACHE_NOHASHDIR fa7ca182a9b ci: Print inner env 84663291275 chain: simplify `deleteRwSettings` code and improve it's doc f8d91f49c70 chain: dont check for null settings value in `overwriteRwSetting` df601993f2d chain: ensure `updateRwSetting` doesn't update to a null settings 5e190cd11f6 Replace CScript _hex_v_u8 appends with _hex cac846c2fbf Allow CScript's operator<< to accept spans, not just vectors c78d8ff4cb8 prevector: avoid GCC bogus warnings in insert method e4e3b44e9cc net: call `Select` with reachable networks in `ThreadOpenConnections` 829becd990b addrman: change `Select` to support multiple networks f698636ec86 net: add `All()` in `ReachableNets` a97f43d63a6 fuzz: Add harness for p2p headers sync c8e2eeeffb4 chain: uniformly use `SettingsAction` enum in settings methods f482d0e366a fuzz: reduce number of iterations in `crypto_aeadchacha20poly1305` target 1e9e735670f chain: move new settings safely in `overwriteRwSetting` 1c409004c80 test: remove wallet context from `write_wallet_settings_concurrently` 58499b00d0a refactor: move `SignSignature` helpers to test utils cd0edf26c07 tracing: cast block_connected duration to nanoseconds cfd03de965a Add Testnet4 launch shortcut for Windows 77b2923f871 Add Signet launch shortcut for Windows bc52cda1f3c fix use int32_t instead of int type for risczero compile with (-march=rv32i, -mabi=ilp32) a0eaa4749fe Add FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION in PoW check a3f6f5acd89 build: Automatically define FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION for fuzz builds 0c02d4b2bdb net_processing: Make MAX_HEADERS_RESULTS a PeerManager option 51f7668d31e addrman: change nid_type from int to int64_t 051ba3290e3 addrman, refactor: introduce user-defined type for internal nId bdad0243be8 rpc, net: getrawaddrman "mapped_as" follow-ups fa1b139d17d Bump python minimum supported version to 3.10 ec585f11c38 Reserve space for transaction inputs in CreateTransactionInternal c76aaaf9003 Reserve space for transaction outputs in CreateTransactionInternal bb3b980dfd9 validation: drop maximum -dbcache REVERT: e70e527ef21 kernel: Add pure kernel bitcoin-chainstate REVERT: ed9a8a54d3c kernel: Add block index utility functions to C header REVERT: 6338dd45b55 kernel: Add function to read block undo data from disk to C header REVERT: a2ac0c1e7c9 kernel: Add functions to read block from disk to C header REVERT: 170060c3372 kernel: Add function for copying block data to C header REVERT: 0f8c00bba07 kernel: Add functions for the block validation state to C header REVERT: 41dba7d2603 kernel: Add validation interface to C header REVERT: 877cf01f22c kernel: Add interrupt function to C header REVERT: f77c2b90422 kernel: Add import blocks function to C header REVERT: 254e17dbeab kernel: Add chainstate load options for in-memory dbs in C header REVERT: 8baa06d318f kernel: Add options for reindexing in C header REVERT: 0243ed8a200 kernel: Add block validation to C header REVERT: 36fbab87e9e Kernel: Add chainstate loading to kernel C header REVERT: b249e93f295 kernel: Add chainstate manager object to C header REVERT: 2546745a393 kernel: Add notifications context option to C header REVERT: 2578746e87f kerenl: Add chain params context option to C header REVERT: 21107de0ca7 kernel: Add kernel library context object REVERT: 83cc65c4911 kernel: Add logging to kernel library C header REVERT: 2f47169f91e kernel: Introduce initial kernel C header API git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 1047757ea3b4b78b51d7338ea44e2123851143fe

sipa changed the title ~~cluster mempool: extend DepGraph (multiple dependencies, removing transactions, parents/children)~~ cluster mempool: extend DepGraph functionality Sep 10, 2024

sipa force-pushed the 202409_clusterlin_partialdep branch from c9620c6 to aae9a83 Compare September 10, 2024 01:32

DrahtBot added the CI failed label Sep 10, 2024

sipa force-pushed the 202409_clusterlin_partialdep branch from aae9a83 to 6c6ecdb Compare September 10, 2024 01:40

sipa force-pushed the 202409_clusterlin_partialdep branch from 6c6ecdb to 9a560f9 Compare September 10, 2024 12:47

sipa force-pushed the 202409_clusterlin_partialdep branch from 9a560f9 to 7510a65 Compare September 10, 2024 15:07

glozow added the Utils/log/libs label Sep 10, 2024

DrahtBot mentioned this pull request Sep 11, 2024

cluster mempool: optimized candidate search #30286

Merged

DrahtBot removed the CI failed label Sep 11, 2024

sipa mentioned this pull request Sep 16, 2024

Cluster mempool tracking issue #30289

Open

22 tasks

DrahtBot added the Needs rebase label Sep 16, 2024

sipa force-pushed the 202409_clusterlin_partialdep branch from 7510a65 to 408d2fd Compare September 16, 2024 20:43

DrahtBot removed the Needs rebase label Sep 16, 2024

sipa force-pushed the 202409_clusterlin_partialdep branch from 408d2fd to 9f67d6d Compare September 17, 2024 13:02

instagibbs reviewed Sep 17, 2024

View reviewed changes

src/cluster_linearize.h Show resolved Hide resolved

DrahtBot mentioned this pull request Sep 17, 2024

Cluster linearization: separate tests from tests-of-tests #30605

Merged

glozow reviewed Sep 17, 2024

View reviewed changes

src/cluster_linearize.h Show resolved Hide resolved

sipa force-pushed the 202409_clusterlin_partialdep branch from 9f67d6d to 4476915 Compare September 18, 2024 17:44

sipa commented Sep 18, 2024

View reviewed changes

sipa force-pushed the 202409_clusterlin_partialdep branch 3 times, most recently from 2b874e3 to 2b21aeb Compare September 30, 2024 22:09

sdaftuar approved these changes Oct 1, 2024

View reviewed changes

src/test/util/cluster_linearize.h Outdated Show resolved Hide resolved

src/cluster_linearize.h Show resolved Hide resolved

src/test/util/cluster_linearize.h Outdated Show resolved Hide resolved

ismaelsadeeq reviewed Oct 4, 2024

View reviewed changes

ismaelsadeeq reviewed Oct 7, 2024

View reviewed changes

src/test/util/cluster_linearize.h Outdated Show resolved Hide resolved

src/cluster_linearize.h Outdated Show resolved Hide resolved

ismaelsadeeq reviewed Oct 7, 2024

View reviewed changes

src/cluster_linearize.h Outdated Show resolved Hide resolved

sipa and others added 7 commits October 7, 2024 13:46

clusterlin: abstract out DepGraph::GetReduced{Parents,Children}

5901cf7

A fuzz test already relies on these operations, and a future commit will need the same logic too. Therefore, abstract them out into proper member functions, with proper testing.

clusterlin: simplify DepGraphFormatter::Ser

abf5064

This does not change the serialization format. It turns out that it is unnecessary to keep track of the order of transactions in the so-far reconstructed DepGraph to decide how far from the end to insert a new transaction.

clusterlin: remove Cluster type

0b3ec8c

sipa force-pushed the 202409_clusterlin_partialdep branch from 2b21aeb to 0b3ec8c Compare October 7, 2024 17:50

DrahtBot added CI failed and removed CI failed labels Oct 7, 2024

instagibbs approved these changes Oct 8, 2024

View reviewed changes

DrahtBot requested review from sdaftuar and ismaelsadeeq October 8, 2024 14:27

ismaelsadeeq reviewed Oct 8, 2024

View reviewed changes

sdaftuar approved these changes Oct 10, 2024

View reviewed changes

glozow reviewed Oct 10, 2024

View reviewed changes

glozow merged commit 489e5aa into bitcoin:master Oct 10, 2024
16 checks passed

		@@ -19,14 +19,6 @@

		namespace cluster_linearize {

		/** Data type to represent cluster input.

cluster mempool: extend DepGraph functionality #30857

cluster mempool: extend DepGraph functionality #30857

Uh oh!

Conversation

sipa commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage

Reviews

Conflicts

Uh oh!

DrahtBot commented Sep 10, 2024

Uh oh!

fanquake commented Sep 10, 2024

Uh oh!

sipa commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

instagibbs commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipa commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipa commented Sep 16, 2024

Uh oh!

instagibbs left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

instagibbs Sep 17, 2024

Choose a reason for hiding this comment

Uh oh!

sipa Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

instagibbs Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

sipa Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

instagibbs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sipa commented Sep 18, 2024

Uh oh!

glozow commented Sep 18, 2024

Uh oh!

sipa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sipa Sep 18, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

instagibbs commented Sep 18, 2024

Uh oh!

sipa commented Sep 10, 2024 •

edited

Loading

DrahtBot commented Sep 10, 2024 •

edited

Loading

sipa commented Sep 10, 2024 •

edited

Loading

instagibbs commented Sep 10, 2024 •

edited

Loading

sipa commented Sep 10, 2024 •

edited

Loading

instagibbs left a comment •

edited

Loading

sipa left a comment •

edited

Loading