dbwrapper: Bump LevelDB max file size to 32 MiB to avoid system slowdown from high disk cache flush rate #30039

maciejsszmigiero · 2024-05-04T09:41:44Z

The default max file size for LevelDB is 2 MiB, which results in the LevelDB compaction code generating ~4 disk cache flushes per second when syncing with the Bitcoin network.
These disk cache flushes are triggered by fdatasync() syscall issued by the LevelDB compaction code when reaching the max file size.

If the database is on a HDD this flush rate brings the whole system to a crawl.
It also results in very slow throughput since 2 MiB * 4 flushes per second is about 8 MiB / second max throughput, while even an old HDD can pull 100 - 200 MiB / second streaming throughput.

Increase the max file size for LevelDB to 128 MiB instead so the flush rate drops to about 1 flush / 2 seconds and the system no longer gets so sluggish.

The max file size value chosen also matches the MAX_BLOCKFILE_SIZE file size setting already used by the block storage.

DrahtBot · 2024-05-04T09:41:47Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30039.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	l0rinc, andrewtoth, TheCharlatan, willcl-ark, tdb3, laanwj, davidgumberg
Concept ACK	sipa

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

sipa · 2024-05-04T14:13:03Z

@jamesob Feel like benchmarking a reindex or so with this?

laanwj · 2024-05-04T16:41:58Z

Are there any drawbacks to this?

willcl-ark · 2024-05-04T16:52:11Z

Might partially address #29662

maciejsszmigiero · 2024-05-04T16:57:21Z

Are there any drawbacks to this?

I didn't notice any.

It's worth mentioning that the total amount of data stored in this database is at least two orders of magnitude higher than even 128 MiB file size.

laanwj · 2024-05-04T17:06:35Z

It's worth mentioning that the total amount of data stored in this database is at least two orders of magnitude higher than even 128 MiB file size.

Oh yes, i asked because i like this even from a "leveldb creates less files" point of view, eg anecdotally on one of my nodes the counter exceeds 6 digits bitcoin-core/bitcoin-maintainer-tools#161 . Of course, this includes deleted files, the active number is "only" about 6000.

tdb3 · 2024-05-05T18:30:41Z

Are there any drawbacks to this?

I didn't notice any.

It's worth mentioning that the total amount of data stored in this database is at least two orders of magnitude higher than even 128 MiB file size.

It would be great if there are few/no drawbacks. Do you mind sharing the methods used so far to test this? It would be great to have some data for comparison.

Other questions that come to mind (thinking out loud before I dig deeper or perform testing):

Does the change from 2MB to 128MB have any impact on consistent or transient RAM usage (i.e. for resource-constrained nodes)?
Is the file size (or an option to use the legacy smaller file size) something we would want to expose in bitcoin.conf (e.g. as a debug option)?

andrewtoth · 2024-05-05T21:31:42Z

Might partially address #29662

That issue is complaining about long compaction times. From https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/include/leveldb/options.h#L111-L112:

The downside will be longer compactions and hence longer latency/performance hiccups.

it seems this change would make compaction times longer, so would exacerbate that issue?

maciejsszmigiero · 2024-05-05T22:51:03Z

Do you mind sharing the methods used so far to test this?

I am simply watching the disk cache flush rate in iostat(1).
In addition to that, the difference in the system interactivity is also pretty apparent.

Does the change from 2MB to 128MB have any impact on consistent or transient RAM usage (i.e. for resource-constrained nodes)?

Did not observe any such effect, the RAM usage of the Bitcoin process seems to vary within roughly the same bounds when syncing with the Bitcoin network with our without this change.

Is the file size (or an option to use the legacy smaller file size) something we would want to expose in bitcoin.conf (e.g. as a debug option)?

Maybe, but I don't know whether it makes sense to expose additional tuning option with respect to, for example, maintenance impact.

The downside will be longer compactions and hence longer latency/performance hiccups.

it seems this change would make compaction times longer, so would exacerbate that issue?

For me, the biggest performance impact of compaction is from disk cache flushes this operation generates.
This patch significantly reduces such flush rate and so should make compaction less painful.

andrewtoth · 2024-05-06T00:55:06Z

src/dbwrapper.cpp

@@ -147,6 +147,7 @@ static leveldb::Options GetOptions(size_t nCacheSize)
        // on corruption in later versions.
        options.paranoid_checks = true;
    }
+    options.max_file_size = 128 << 20;


Should we make this a constant? Would it be appropriate to reuse MAX_BLOCKFILE_SIZE?

+1 on a constant, but i don't think it's approprioate to reuse MAX_BLOCKFILE_SIZE, better to define a new one

Added a relevant constant.

andrewtoth · 2024-05-06T02:18:12Z

Benchmarked IBD with an SSD to block 800k, dbcache=450, prune=0 with a local node serving the blocks. This branch is 27% (!) faster than master 🚀

 commit 7f15e71f7e762645dbd1ea5eba9ecc6f9ad60236 (branch)
  Time (mean ± σ):     14711.490 s ± 225.376 s    [User: 19465.517 s, System: 1147.712 s]
  Range (min … max):   14552.125 s … 14870.854 s    2 runs
  
 commit eb0bdbdd753bca97120247b921fd29d606fea6e9 (master)
  Time (mean ± σ):     20274.276 s ± 106.042 s    [User: 21762.310 s, System: 4546.936 s]
  Range (min … max):   20199.293 s … 20349.259 s    2 runs

This patch significantly reduces such flush rate and so should make compaction less painful.

From what I understand, this patch reduces the frequency of flushes, but they will take longer when they do occur. This is great for IBD, but for #29662 the issue is an unavoidable compaction at startup. The compaction could potentially take longer with this patch.

laanwj · 2024-05-06T07:56:25Z

This branch is 27% (!) faster than master

That's impressive!

From what I understand, this patch reduces the frequency of flushes

Not only the frequency of flushes; another potential advantage here is that leveldb will spend less time open()ing and close()ing files to maintain its allowed number of open files (eg the fd_limiter stuff).

luke-jr · 2024-05-07T17:49:06Z

If there's no drawbacks, why not go even larger?

willcl-ark · 2024-05-07T18:38:15Z

I ran some benchmarks of IBD to block 800,000 vs master for comparison, and got some similar, if slightly less impressive, results with default dbcache.

With -dbcache=16384:

master@ fdb41e0: 9607 s
master@ fdb41e0 + 7f15e71: 9351 s
3% faster with this change

With -dbcache=450:

master@ fdb41e0: 15338 s
master@ fdb41e0 + 7f15e71: 13246 s
~16% faster with this change

I only did a single run of each though. Sync was performed from a single second local node with datadir on a separate SSD.

andrewtoth · 2024-05-07T19:18:03Z

FWIW re: #29662 I did not notice any difference in compaction time at startup on an SSD. It takes about 5 seconds to finish with debug=leveldb both on master and this branch.

maciejsszmigiero · 2024-05-07T22:07:51Z

If there's no drawbacks, why not go even larger?

I used 128 MiB as the new size for commonality with MAX_BLOCKFILE_SIZE already used by the block storage and because it gives me a nice low disk cache flush rate of about 1 flush / 2 seconds that no longer impacts the overall system performance.

But just to be sure, changed the patch to use std::max() around this max_file_size option so if at some point LevelDB decides to increase its default above 128 MiB we won't be lowering it accidentally.

mzumsande

I've played around with this branch a bit, upgrading and downgrading between it and master with existing datadirs on signet and didn't run into any issues.
Also just noting that this will affect all leveldb databases, also the indexes and the block/index db.

sipa · 2024-05-17T18:23:58Z

It appears that RocksDb (more-developed derivative of LevelDB) uses a default of 64 MiB (https://github.com/facebook/rocksdb/blob/main/include/rocksdb/advanced_options.h#L468). See also my comment in #30059 (comment).

l0rinc · 2024-10-02T12:18:45Z

I did a few benchmarks on HDD and SSD separately (no raspberry pi yet, but I understood @davidgumberg did some of those and saw a significant speedup), to see the effect of the different values on IBD.

I have tried different values via #30059 (rebased), namely 1,2,4,8,16,32,64,128,256,512 MiB (current value is 2) with default dbcache, until 600k blocks using real nodes (which introduces some randomness, but the repeated runs should still indicate a trend).

benchmark

hyperfine \
  --runs 1 \
  --export-json /mnt/my_storage/ibd_benchmark.json \
  --parameter-list DBFILESIZE 1,2,4,8,16,32,64,128,256,512 \
  --prepare 'rm -rf /mnt/my_storage/BitcoinData/*' \
  './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize={DBFILESIZE} -printtoconsole=0'

SSD:

Benchmark 1: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=1 -printtoconsole=0
  Time (abs ≡):        9376.982 s               [User: 8939.258 s, System: 2037.366 s]

Benchmark 2: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=2 -printtoconsole=0
  Time (abs ≡):        7809.227 s               [User: 8399.808 s, System: 1258.152 s]

Benchmark 3: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=4 -printtoconsole=0
  Time (abs ≡):        7060.817 s               [User: 8210.950 s, System: 626.069 s]

Benchmark 4: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=8 -printtoconsole=0
  Time (abs ≡):        7201.632 s               [User: 8046.769 s, System: 615.964 s]

Benchmark 5: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=16 -printtoconsole=0
  Time (abs ≡):        7848.417 s               [User: 8394.320 s, System: 713.182 s]

Benchmark 6: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=32 -printtoconsole=0
  Time (abs ≡):        8289.161 s               [User: 8183.729 s, System: 599.698 s]

Benchmark 7: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=64 -printtoconsole=0
  Time (abs ≡):        7580.532 s               [User: 8077.446 s, System: 612.879 s]

Benchmark 8: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=128 -printtoconsole=0
  Time (abs ≡):        9060.371 s               [User: 8140.057 s, System: 606.641 s]

Benchmark 9: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=256 -printtoconsole=0
  Time (abs ≡):        8778.117 s               [User: 8001.854 s, System: 620.595 s]

Benchmark 10: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=512 -printtoconsole=0
  Time (abs ≡):        7856.151 s               [User: 7970.946 s, System: 680.476 s]

Summary
  './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=4 -printtoconsole=0' ran
    1.02 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=8 -printtoconsole=0'
    1.07 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=64 -printtoconsole=0'
    1.11 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=2 -printtoconsole=0'
    1.11 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=16 -printtoconsole=0'
    1.11 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=512 -printtoconsole=0'
    1.17 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=32 -printtoconsole=0'
    1.24 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=256 -printtoconsole=0'
    1.28 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=128 -printtoconsole=0'
    1.33 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=500000 -dbfilesize=1 -printtoconsole=0'

and HDD

Benchmark 1: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=1 -printtoconsole=0
  Time (abs ≡):        10150.860 s               [User: 8046.261 s, System: 1557.130 s]

Benchmark 2: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=2 -printtoconsole=0
  Time (abs ≡):        8935.037 s               [User: 7746.422 s, System: 981.186 s]

Benchmark 3: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=4 -printtoconsole=0
  Time (abs ≡):        7636.675 s               [User: 7348.012 s, System: 547.172 s]

Benchmark 4: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=8 -printtoconsole=0
  Time (abs ≡):        7633.078 s               [User: 7306.267 s, System: 572.424 s]

Benchmark 5: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=16 -printtoconsole=0
  Time (abs ≡):        7639.829 s               [User: 7266.532 s, System: 591.955 s]

Benchmark 6: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=32 -printtoconsole=0
  Time (abs ≡):        7345.802 s               [User: 7265.908 s, System: 584.797 s]

Benchmark 7: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=64 -printtoconsole=0
  Time (abs ≡):        7617.101 s               [User: 7092.537 s, System: 551.785 s]

Benchmark 8: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=128 -printtoconsole=0
  Time (abs ≡):        7508.948 s               [User: 7065.206 s, System: 580.337 s]

Benchmark 9: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=256 -printtoconsole=0
  Time (abs ≡):        7563.822 s               [User: 7093.650 s, System: 599.636 s]

Benchmark 10: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=512 -printtoconsole=0
  Time (abs ≡):        7600.085 s               [User: 6997.129 s, System: 536.973 s]

Summary
  ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=32 -printtoconsole=0 ran
    1.02 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=128 -printtoconsole=0
    1.03 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=256 -printtoconsole=0
    1.03 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=512 -printtoconsole=0
    1.04 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=64 -printtoconsole=0
    1.04 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=8 -printtoconsole=0
    1.04 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=4 -printtoconsole=0
    1.04 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=16 -printtoconsole=0
    1.22 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=2 -printtoconsole=0
    1.38 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=500000 -dbfilesize=1 -printtoconsole=0

While these measurements aren't definitive, both hinted at -dbfilesize=4 being better than -dbfilesize=2 (the default) and may not be a lot better than -dbfilesize=128.

I'll rerun these with 2,4,8,64,128 and 800k blocks on the HDD to validate the findings.

davidgumberg · 2024-10-04T19:10:54Z

I cherry picked your branch onto master and did two runs syncing from a stable, dedicated local node twice on a Raspberry Pi 5 4GB using microSD for storage, with a prune of 2000 and the default dbcache using the following command:

./build/src/bitcoind -daemon=0 -connect=ryzen7900xnode:8333 -stopatheight=800000 -prune=2000 -debug=bench -debug=blockstorage -debug=coindb -debug=mempool -debug=prune

I saw a massive improvement, with your branch taking, on average, ~67.8% of the time taken by the master branch to reach block height 800,000¹:

	Avg (hh:mm:ss)	Run 1	Run 2
Master	47:17:14 (170,234s)	48:38:05 (175,085s)	45:56:22 (165,382s)
Branch, cherry picked onto master	32:01:14 (115,274s)	34:06:26 (122,786s)	29:56:01 (107,761s)

For me this validates that a substantial performance improvement is possible. I suspect especially on disk I/O constrained setups, and I'm really interested in making IBD on Raspberry Pi's faster.

Concept ACK on looking into the tradeoffs of different settings here. Not to try and duplicate discussion too much between this and #30059, but I second @l0rinc that looking for one good default seems better than making this configurable, unless we find evidence that different setups benefit substantially from different values.

But, I think more work needs to be done to identify what value works best here, and hopefully come up with an account for why, I will try to bench some different max file size values on the Raspberry Pi that I have similar to @l0rinc's work above.

These benchmarks took so long that the weather had changed between run 1 and run 2, and I am not running these in a room where the temperature is very well controlled which I believe is the primary cause of run 2 being faster for both. ↩

l0rinc · 2024-10-07T15:20:11Z

Finished benchmarking with the default 2 mb file size vs 4, 8, 64 and 128 mb.
This time it's full IBD with real peers until 800k blocks on a HDD.

Benchmarks

hyperfine   --runs 1   --export-json /mnt/ibd_DBFILESIZE.json   --parameter-list DBFILESIZE 2,4,8,64,128   --prepare 'rm -rf /mnt/BitcoinData/*'   './build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize={DBFILESIZE} -printtoconsole=0'

2, 4, 8, 64, 128

 Benchmark 1: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=2 -printtoconsole=0
  Time (abs ≡):        36403.630 s               [User: 31186.459 s, System: 5761.138 s]

 Benchmark 2: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=4 -printtoconsole=0
  Time (abs ≡):        30540.101 s               [User: 29188.931 s, System: 3430.547 s]

Benchmark 3: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=8 -printtoconsole=0
  Time (abs ≡):        28913.948 s               [User: 28857.575 s, System: 2292.117 s]

Benchmark 4: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=64 -printtoconsole=0
  Time (abs ≡):        27911.380 s               [User: 28268.729 s, System: 2179.778 s]

Benchmark 5: ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=128 -printtoconsole=0
  Time (abs ≡):        28191.359 s               [User: 27915.963 s, System: 2045.088 s]

Summary

  ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=64 -printtoconsole=0 ran
    1.01 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=128 -printtoconsole=0
    1.04 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=8 -printtoconsole=0
    1.09 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=4 -printtoconsole=0
    1.30 times faster than ./build/src/bitcoind -datadir=/mnt/BitcoinData -stopatheight=800000 -dbfilesize=2 -printtoconsole=0

Edit:

Repeated the same for SSD, very similar results:

benchmark

hyperfine   --runs 1   --export-json /mnt/ibd_DBFILESIZE-ssd.json   --parameter-list DBFILESIZE 2,8,16,32,64 --prepare 'rm -rf /mnt/my_storage/BitcoinData/*'  './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize={DBFILESIZE} -printtoconsole=0 -dbcache=1000'

2, 8, 16, 32, 64

Benchmark 1: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=2 -printtoconsole=0 -dbcache=1000
  Time (abs ≡):        32323.964 s               [User: 30174.040 s, System: 6349.312 s]
 
Benchmark 2: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=8 -printtoconsole=0 -dbcache=1000
  Time (abs ≡):        24513.755 s               [User: 27618.551 s, System: 1728.897 s]
 
Benchmark 3: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=16 -printtoconsole=0 -dbcache=1000
  Time (abs ≡):        24648.438 s               [User: 27925.669 s, System: 1893.671 s]
 
Benchmark 4: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=32 -printtoconsole=0 -dbcache=1000
  Time (abs ≡):        24797.871 s               [User: 27621.893 s, System: 1755.004 s]
 
Benchmark 5: ./build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=64 -printtoconsole=0 -dbcache=1000
  Time (abs ≡):        25078.417 s               [User: 27879.669 s, System: 2064.851 s]

Summary

  './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=8 -printtoconsole=0 -dbcache=1000' ran
    1.01 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=16 -printtoconsole=0 -dbcache=1000'
    1.01 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=32 -printtoconsole=0 -dbcache=1000'
    1.02 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=64 -printtoconsole=0 -dbcache=1000'
    1.32 times faster than './build/src/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=800000 -dbfilesize=2 -printtoconsole=0 -dbcache=1000'

In conclusion it seems to me that 2mb is indeed too low, there seems to be a significant jump when doubling the file size (~20-30%% faster), but after that the advantage is smaller (8mb is 25% faster, 64 mb is 30% faster and 128 is 29% faster).

Since we're not yet sure of all the second order effects of this change (longer compaction, more memory, migration problems, etc), I wouldn't yet recommend jumping to 128, but to 8 or 16 only.

src/dbwrapper.cpp

l0rinc · 2024-11-06T20:54:41Z

@maciejsszmigiero, are you still working on this or should we take over?

I can also confirm that it's possible to just switch file size values back-and-forth without needing a reindex.
I have reindexed until block 600k with master vs 16 mb blocks (instead of the 128 for the reasons mentioned before).

The LevelDB files seem to effortlessly change from 2 mb to 17 :

chainstate/062435.ldb - 906'412 bytes
chainstate/061885.ldb - 2'171'330 bytes
chainstate/063212.ldb - 1'936'570 bytes
chainstate/064711.ldb - 982'165 bytes
chainstate/061518.ldb - 2'171'520 bytes
chainstate/062708.ldb - 2'169'653 bytes
chainstate/061659.ldb - 2'171'631 bytes
chainstate/063237.ldb - 2'170'487 bytes
→
chainstate/062435.ldb - 906'412 bytes
chainstate/065302.ldb - 17'347'086 bytes
chainstate/062708.ldb - 2'169'653 bytes
chainstate/063237.ldb - 2'170'487 bytes

And when reverting to master, effortlessly go back:

chainstate/062468.ldb - 2'171'399 bytes
chainstate/065305.ldb - 17'358'270 bytes
chainstate/062543.ldb - 2'172'244 bytes
→
chainstate/062468.ldb - 2'171'399 bytes
chainstate/065579.ldb - 2'170'605 bytes
chainstate/068994.ldb - 2'169'617 bytes
chainstate/068954.ldb - 2'170'158 bytes
chainstate/062543.ldb - 2'172'244 bytes

The total bytes on disk seems to be basically the same, but the number of files is reduced considerably (might alleviate open file problems):

before 2168 files, 4'383'947'229 bytes
after 280 files, 4'386'553'693 bytes

maciejsszmigiero · 2024-11-09T21:40:20Z

@l0rinc

are you still working on this or should we take over?

I can obviously change the default in this PR to 16 MiB but I think having #30059 is important too: as you measured here on Oct 2 the best performing size on HDD storage actually seems to be 32 MiB.

willcl-ark · 2024-11-22T10:30:09Z

I would also support slightly reducing the value in this PR, my preference though would be 32MB, for these reasons:

The benchmark data shows the biggest gains come from the initial increases (2MB → 4MB → 8MB)
There are diminishing returns after 8MB, with 128MB actually performing slightly worse than 64MB in total time in some of the benchmarks above
Most of the performance gains are captured by the 32MB size, esp. when including HDDs
Smaller files will also:
- Be more manageable in memory-constrained environments (relevant for our current default 450MB cache)
- Create less memory pressure during compaction operations*
- Allow for more granular cache utilization (unclear to me if/how this affects us though, I've not measured this)

* If I am understanding LevelDB compaction correctly, the 32MB filesize will use less peak memory during merge operations than a 128MB filesize, which would be useful for our resource-constrained use cases (Raspberry Pi nodes).

The system time improvements in the benchmarks suggest we'll get most of the benefits at 32MB, while keeping better compatibility with our default memory settings.

willcl-ark

ACK b73d331

tdb3

ACK b73d331

laanwj

ACK b73d331

davidgumberg · 2024-12-01T05:28:12Z

ACK b73d331

…7c2ccc920 997c2ccc920 Add hooks for script debug callbacks 8598bc9e5d3 kernel: Add pure kernel bitcoin-chainstate 07530efd10c kernel: Add functions to get the block hash from a block 1a871d96afd kernel: Add block index utility functions to C header a50a97dec8e kernel: Add function to read block undo data from disk to C header 093489acee8 kernel: Add functions to read block from disk to C header 8a4b4fdefab kernel: Add function for copying block data to C header b09983496ae kernel: Add functions for the block validation state to C header c445389d749 kernel: Add validation interface to C header 94f65e6fc8c kernel: Add interrupt function to C header a7eac722561 kernel: Add import blocks function to C header c6a7574247f kernel: Add chainstate load options for in-memory dbs in C header 6f7bb78869c kernel: Add options for reindexing in C header 46f5909e6d8 kernel: Add block validation to C header 1680886ba03 Kernel: Add chainstate loading to kernel C header 4aad77d2665 kernel: Add chainstate manager option for setting worker threads c92849c7b51 kernel: Add chainstate manager object to C header e420bbddf36 kernel: Add notifications context option to C header d3c6127e0d3 kerenl: Add chain params context option to C header 4bc8c494693 kernel: Add kernel library context object df66c7b0ff2 kernel: Add logging to kernel library C header a78aeb9ff0e kernel: Introduce initial kernel C header API 6cd95de2e02 Merge bitcoin/bitcoin#31395: build: Set shared linker flags in toolchain file abeebccc480 Merge bitcoin/bitcoin#31357: cmake: Improve build script correctness 4c9b13841c4 Merge bitcoin/bitcoin#31402: doc: correct libfuzzer-nosan preset flag da4f4fac8df Merge bitcoin/bitcoin#31361: cmake, qt: Use absolute paths for includes in MOC-generated files 16b140f225a doc: correct libfuzzer-nosan preset flag 097c66f6148 Merge bitcoin/bitcoin#30039: dbwrapper: Bump LevelDB max file size to 32 MiB to avoid system slowdown from high disk cache flush rate 68daaea0e48 Merge bitcoin/bitcoin#31390: Remove `src/config` directory 14f162dc5c8 Merge bitcoin/bitcoin#31399: ci, macos: Install `pkgconf` Homebrew's package e2f2698395c ci, macos: Install `pkgconf` Homebrew's package b73d3319377 dbwrapper: Bump max file size to 32 MiB a8e04704f93 build: Set shared linker flags in toolchain file dbc8ba12f3b Merge bitcoin/bitcoin#31371: doc, test: more ephemeral dust follow-ups 935973b315f Remove `src/config` directory 7590e93bc73 Merge bitcoin/bitcoin#30986: contrib: skip missing binaries in gen-manpages b2af068825c Merge bitcoin/bitcoin#30708: rpc: add getdescriptoractivity 144f98db85e Merge bitcoin/bitcoin#31337: build: Fix coverage builds efdb49afb9e Merge bitcoin/bitcoin#31323: guix: swap `moreutils` for just `sponge` 37a5c5d8366 doc: update descriptors.md for getdescriptoractivity ee3ce6a4f4d test: rpc: add no address case for getdescriptoractivity 811f76f3a51 rpc: add getdescriptoractivity ee6185372fc gen-manpages: Prompt error if no binaries are found 70e20ea024c Merge bitcoin/bitcoin#31172: build: increase minimum supported Windows to 10.0 733317ba943 Merge bitcoin/bitcoin#31364: refactor: Fix remaining clang-tidy performance-unnecessary-copy-initialization errors 5a4bc5c0366 Merge bitcoin/bitcoin#31305: refactor: Fix remaining clang-tidy performance-inefficient-vector errors 28fd0bc7316 Merge bitcoin/bitcoin#31365: interpreter: Use the same type for SignatureHash in the definition 72ab35a6d09 Merge bitcoin/bitcoin#31221: ci: Split out native fuzz jobs for macOS and windows (take 2) 160799d9135 test: refactor: introduce `create_ephemeral_dust_package` helper 61e18dec306 doc: ephemeral policy: add missing closing double quote 3305972f7bf refactor: Fix remaining clang-tidy performance-unnecessary-copy-initialization errors 11f3bc229cc refactor: Reserve vectors in fuzz tests 152fefe7a22 refactor: Preallocate PrevectorFillVector(In)Direct without vector resize a774c7a339c refactor: Fix remaining clang-tidy performance-inefficient-vector errors f7144b24be0 Merge bitcoin/bitcoin#31279: policy: ephemeral dust followups c288c790cd9 interpreter: Use the same type for SignatureHash in the definition b031b7910d6 [ci] Split out native fuzz jobs for macOS and windows 6f4128e3a83 cmake, qt: Use absolute paths for includes in MOC-generated files ab5c63edcce cmake: Build `secp256k1` only when required 76a3a540a4c cmake: Ensure script correctness when no targets are specified e8f50c5debe guix: swap moreutils for just sponge 01a7298818d build: Avoid using the `-ffile-prefix-map` compiler option 2638fdb4f93 Merge bitcoin/bitcoin#31338: test: Deduplicate assert_mempool_contents() 17834bd1976 Merge bitcoin/bitcoin#31333: fuzz: Implement G_TEST_GET_FULL_NAME cf577227888 Merge bitcoin/bitcoin#31335: macOS: swap docs & CI from pkg-config to pkgconf fe3457ccfff ci: note that we should install pkgconf in future a0eafc10f94 functional test: Deduplicate assert_mempool_contents() 8d203480b33 doc: migrate from pkg-config to pkgconf in macOS build docs 466e4df3fb8 assert_mempool_contents: assert not duplicates expected ea5db2f2692 functional: only generate required blocks for test d033acb6083 fuzz: package_eval: let fuzzer run out input in main tx creation loop ba35a570c5d CheckEphemeralSpends: return boolean, and set child state and txid outparams cf0cee1617c func: add note about lack of 1P1C propagation in tree submitpackage 84242903043 unit test: ephemeral_tests is using a dust relay rate, not minrelay d9cfa5fc4eb CheckEphemeralSpends: no need to iterate inputs if no parent dust 87b26e3dc07 func: rename test_free_relay to test_no_minrelay_fee e5709a4a41e func: slight elaboration on submitpackage restriction 08e969bd107 RPC: only enforce dust rules on priority when standardness active ca050d12e76 unit test: adapt to changing MAX_DUST_OUTPUTS_PER_TX 7c3490169c9 fuzz: package_eval: move last_tx inside txn ctor 445eaed182a fuzz: use optional status instead of should_rbf_eph_spend 4dfdf615b9d fuzz: remove unused TransactionsDelta validation interface 09ce926e4a1 func: cleanup reorg test comment 768a0c1889e func: cleanup test_dustrelay comments bedca1cb663 fuzz: Directly place transactions in vector c041ad6eccb fuzz: explain package eval coin tracking better bc0d98ea612 fuzz: remove dangling reference to GetEntry 15b6cbf07f5 unit test: make dust index less magical 5fbcfd12b8f unit test: assert txid returned on CheckEphemeralSpends failures ef94d84b4e4 bench: remove unnecessary CMTxn constructors c5c10fd317c ephemeral policy doxygen cleanup dd9044b8d46 ephemeral policy: IWYU c6859ce2de7 Move+rename GetDustIndexes -> GetDust 22ef95dbe3e Merge bitcoin/bitcoin#31288: Add destroy to BlockTemplate schema 92d3d691f09 fuzz: Implement G_TEST_GET_FULL_NAME f34fe0806a0 Merge bitcoin/bitcoin#31122: cluster mempool: Implement changeset interface for mempool b2d952c0f5b Merge bitcoin/bitcoin#31331: doc: add copyright header to p2p_headers_presync 7d3703dec3d doc: add copyright header to p2p_headers_presync 116b8c55736 Merge bitcoin/bitcoin#31213: fuzz: Fix difficulty target generation in `p2p_headers_presync` 15c1f47a005 Merge bitcoin/bitcoin#31327: doc: Correct PR Review Club frequency from weekly to monthly 1209a1082c8 Merge bitcoin/bitcoin#31315: build: Enable -Wbidi-chars=any ab22726def9 Merge bitcoin/bitcoin#31276: guix: scope pkg-config to Linux only 637f437a164 doc: remove PR Review Club frequency e1223099584 Merge bitcoin/bitcoin#31317: test: Revert to random path element 2666d83da59 Merge bitcoin/bitcoin#30893: test: Introduce ensure_for helper faaaf59f71e test: Make g_rng_temp_path rand, not dependent on SeedRandomForTest 746f93b4f0f Merge bitcoin/bitcoin#31307: build: Temporarily disable compiling `fuzz/utxo_snapshot.cpp` with MSVC 25fe087de59 rpc: move-only: move ScriptPubKeyDoc to utils fa80b08fef0 test: Revert to random path element fa7857ccda5 build: Enable -Wbidi-chars=any b2d53610028 build: Temporarily disable compiling `fuzz/utxo_snapshot.cpp` with MSVC 9aa50152c1c Add destroy to BlockTemplate schema ccc2d3abcd3 Merge bitcoin/bitcoin#31287: refactor: Avoid std::string format strings 62016b32300 Use std::ranges for ephemeral policy checks 3ed930a1f41 Have HasDust and PreCheckValidEphemeralTx take CTransaction 04a614bf9a7 Rename CheckValidEphemeralTx to PreCheckEphemeralTx 85bcfeea235 Merge bitcoin/bitcoin#30666: validation: fix m_best_header tracking and BLOCK_FAILED_CHILD assignment 2257c6d68fa Merge bitcoin/bitcoin#30487: ci: skip Github CI on branch pushes for forks 380e1f44e8e Merge bitcoin/bitcoin#30349: benchmark: Improve SipHash_32b accuracy to avoid potential optimization issues 1a8f51e7453 Merge bitcoin/bitcoin#28843: [refactor] Cleanup BlockAssembler mempool usage 2d944e982c4 Merge bitcoin/bitcoin#31285: guix: remove `util-linux` bcd82b13f46 Remove pkgconfig from toolchain file 319a4e82614 depends: drop sqlite pkgconfig file fa1177e3d7c refactor: Avoid std::string format strings a8fe1fd38bf depends: better cleanup after fontconfig 17e79c92607 depends: fully remove libtool archives from Qt build 8ca85651c83 guix: move pkg-config to Linux builds e3e648cf410 depends: drop pkg-config option from Qt build 0d185bd99f9 doc: update depends doc to prefer .cmake outputs e546b4e1a0c Merge bitcoin/bitcoin#31225: doc: Fix grammatical errors in multisig-tutorial.md f44e39c9d0d Merge bitcoin/bitcoin#31174: tinyformat: Add compile-time checking for literal format strings 299e2220e95 gen-manpages: implement --skip-missing-binaries 5736d1ddacc tracing: pass if replaced by tx/pkg to tracepoint a4ec07f1944 doc: add comments for CTxMemPool::ChangeSet 83f814b1d11 Remove m_all_conflicts from SubPackageState d3c8e7dfb63 Ensure that we don't add duplicate transactions in rbf fuzz tests d7dc9fd2f7b Move CalculateChunksForRBF() to the mempool changeset 284a1d33f1d Move prioritisation into changeset 446b08b599b Don't distinguish between direct conflicts and all conflicts when doing cluster-size-2-rbf checks b53041021ab Duplicate transactions are not permitted within a changeset b447416fddc Public mempool removal methods Assume() no changeset is outstanding 2b30f4d36c8 Make RemoveStaged() private 18829194ca6 Enforce that there is only one changeset at a time 7fb62f7db60 Apply mempool changeset transactions directly into the mempool 34b6c5833d1 Clean up FinalizeSubpackage to avoid workspace-specific information 57983b8add7 Move LimitMempoolSize to take place outside FinalizeSubpackage 01e145b9758 Move changeset from workspace to subpackage 802214c0832 Introduce mempool changesets 87d92fa3401 test: Add unit test coverage of package rbf + prioritisetransaction 15d982f91e6 Add package hash to package-rbf log message 4d668549825 ci: remove util-linux from centos CI cdf34be7c94 guix: remove util-linux cbf1a47d606 CheckEphemeralSpends: only compute txid of tx when needed 111465d72dd test: Remove unused attempts parameter from wait_until 5468a23eb9a test: Add check_interval parameter to wait_until 16c87d91fd4 test: Introduce ensure_for helper a6ca8f32439 fuzz: Fix difficulty target generation in p2p_headers_presync 8610bcef9d0 ci: skip Github CI on branch pushes for forks ee1128ead84 doc: update stack-clash-protection comment re mingw-w64 bf47448f152 test: drop check for Windows < 10 35b898c47f8 release: target Windows 10 or later 398754e70bc depends: target Windows 10 when building for mingw-w64 ac286e0d1bd doc: Fix grammatical errors in multisig-tutorial.md fa327c77e34 util: Add ConsumeArithUInt256InRange fuzzing helper 42066f45ff5 Refactor SipHash_32b benchmark to improve accuracy and avoid optimization issues fe39acf88ff tinyformat: Add compile-time checking for literal format strings 184f34f2d0f util: Support dynamic width & precision in ConstevalFormatString 192dac1d337 [refactor] Cleanup BlockAssembler mempool usage 0bd53d913c1 test: add test for getchaintips behavior with invalid chains ccd98ea4c88 test: cleanup rpc_getchaintips.py f5149ddb9b7 validation: mark blocks building on an invalid block as BLOCK_FAILED_CHILD 783cb7337f7 validation: call RecalculateBestHeader in InvalidChainFound 9275e9689a4 rpc: call RecalculateBestHeader as part of reconsiderblock a51e91783aa validation: add RecalculateBestHeader() function REVERT: 403c20980ec kernel: Add pure kernel bitcoin-chainstate REVERT: 2e1b262eff6 kernel: Add functions to get the block hash from a block REVERT: f0b56a6052a kernel: Add block index utility functions to C header REVERT: 9393cf98179 kernel: Add function to read block undo data from disk to C header REVERT: e2ccfba0e3e kernel: Add functions to read block from disk to C header REVERT: 0559575861c kernel: Add function for copying block data to C header REVERT: 7b02e52237e kernel: Add functions for the block validation state to C header REVERT: fdebacca7ad kernel: Add validation interface to C header REVERT: f4ea5f49c6d kernel: Add interrupt function to C header REVERT: 5f4b436aad9 kernel: Add import blocks function to C header REVERT: c95a28fd80c kernel: Add chainstate load options for in-memory dbs in C header REVERT: d6360557ef6 kernel: Add options for reindexing in C header REVERT: a125867b9fe kernel: Add block validation to C header REVERT: b2b75a0ef73 Kernel: Add chainstate loading to kernel C header REVERT: d233003ff2a kernel: Add chainstate manager option for setting worker threads REVERT: 3610be3b138 kernel: Add chainstate manager object to C header REVERT: c194bea41f6 kernel: Add notifications context option to C header REVERT: 691d89d846b kerenl: Add chain params context option to C header REVERT: 407ca750cda kernel: Add kernel library context object REVERT: ee3c4ea92cd kernel: Add logging to kernel library C header REVERT: e6c610a7e03 kernel: Introduce initial kernel C header API git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 997c2ccc920adb0d1c9d8e82dbe94374cfb3c78b

…90df267df 6090df267df kernel: Add pure kernel bitcoin-chainstate 29e3b874303 kernel: Add functions to get the block hash from a block 97d83063cb6 kernel: Add block index utility functions to C header 76ead0878a3 kernel: Add function to read block undo data from disk to C header 94c215c4212 kernel: Add functions to read block from disk to C header aff05fcbe62 kernel: Add function for copying block data to C header a379fbe15eb kernel: Add functions for the block validation state to C header 81db1665213 kernel: Add validation interface to C header 0ee7a5f58a6 kernel: Add interrupt function to C header f68b3dbf919 kernel: Add import blocks function to C header 503aae9afc5 kernel: Add chainstate load options for in-memory dbs in C header fa1335fd809 kernel: Add options for reindexing in C header 0eb020f9c82 kernel: Add block validation to C header 5f471729674 Kernel: Add chainstate loading to kernel C header 90949cbb09d kernel: Add chainstate manager option for setting worker threads a2021b517c6 kernel: Add chainstate manager object to C header 428ea46909b kernel: Add notifications context option to C header b1f5d450323 kerenl: Add chain params context option to C header 7bb4b412db0 kernel: Add kernel library context object e4b63d7f5d9 kernel: Add logging to kernel library C header 47bb56b243e kernel: Introduce initial kernel C header API ff873a20a7f Merge bitcoin/bitcoin#31313: refactor: Clamp worker threads in ChainstateManager constructor c9a7418a8df Merge bitcoin/bitcoin#31096: Package validation: accept packages of size 1 6f24662eb96 Merge bitcoin/bitcoin#31175: rpc: Remove submitblock pre-checks 3867d2421ae Merge bitcoin/bitcoin#31112: Improve parallel script validation error debug logging 8e02b480591 Merge bitcoin/bitcoin#31284: ci: Skip broken Wine64 tests by default 492e1f09943 [validation] merge all ConnectBlock debug logging code paths b49df703f03 [validation] include all logged information in BlockValidationState 7b267c034fd [validation] Add detailed txin/txout information for script error messages 146a3d54268 [validation] Make script error messages uniform for parallel/single validation 1ac1c33f3f1 [checkqueue] support user-defined return type through std::optional ebe4cac38bf Merge bitcoin/bitcoin#30991: test: enable running independent functional test sub-tests 19276741007 Merge bitcoin/bitcoin#31387: doc: Use more precise anchor link to codesigning docs e043618d44d Merge bitcoin/bitcoin#31396: test: simple reordering to reduce run time a25b892ab1f Merge bitcoin/bitcoin#31386: doc: Use more precise anchor links to Xcode SDK extraction eb646111cdc Merge bitcoin/bitcoin#31383: test: Add missing node.setmocktime(self.mocktime) to p2p_ibd_stalling.py 6cd95de2e02 Merge bitcoin/bitcoin#31395: build: Set shared linker flags in toolchain file abeebccc480 Merge bitcoin/bitcoin#31357: cmake: Improve build script correctness 4c9b13841c4 Merge bitcoin/bitcoin#31402: doc: correct libfuzzer-nosan preset flag da4f4fac8df Merge bitcoin/bitcoin#31361: cmake, qt: Use absolute paths for includes in MOC-generated files 16b140f225a doc: correct libfuzzer-nosan preset flag 097c66f6148 Merge bitcoin/bitcoin#30039: dbwrapper: Bump LevelDB max file size to 32 MiB to avoid system slowdown from high disk cache flush rate 68daaea0e48 Merge bitcoin/bitcoin#31390: Remove `src/config` directory 14f162dc5c8 Merge bitcoin/bitcoin#31399: ci, macos: Install `pkgconf` Homebrew's package e2f2698395c ci, macos: Install `pkgconf` Homebrew's package b73d3319377 dbwrapper: Bump max file size to 32 MiB 62f6d9e1a48 test: simple ordering optimization to reduce runtime a8e04704f93 build: Set shared linker flags in toolchain file dbc8ba12f3b Merge bitcoin/bitcoin#31371: doc, test: more ephemeral dust follow-ups 935973b315f Remove `src/config` directory 19f49c7489d doc: Use more precise anchor link to codesigning docs 8bf1b3039cb doc: Use more precise anchor links to Xcode SDK extraction 7590e93bc73 Merge bitcoin/bitcoin#30986: contrib: skip missing binaries in gen-manpages b2af068825c Merge bitcoin/bitcoin#30708: rpc: add getdescriptoractivity 144f98db85e Merge bitcoin/bitcoin#31337: build: Fix coverage builds faa16ed4b9e test: Add missing node.setmocktime(self.mocktime) to p2p_ibd_stalling.py efdb49afb9e Merge bitcoin/bitcoin#31323: guix: swap `moreutils` for just `sponge` 37a5c5d8366 doc: update descriptors.md for getdescriptoractivity ee3ce6a4f4d test: rpc: add no address case for getdescriptoractivity 811f76f3a51 rpc: add getdescriptoractivity ee6185372fc gen-manpages: Prompt error if no binaries are found 70e20ea024c Merge bitcoin/bitcoin#31172: build: increase minimum supported Windows to 10.0 733317ba943 Merge bitcoin/bitcoin#31364: refactor: Fix remaining clang-tidy performance-unnecessary-copy-initialization errors 5a4bc5c0366 Merge bitcoin/bitcoin#31305: refactor: Fix remaining clang-tidy performance-inefficient-vector errors 28fd0bc7316 Merge bitcoin/bitcoin#31365: interpreter: Use the same type for SignatureHash in the definition 72ab35a6d09 Merge bitcoin/bitcoin#31221: ci: Split out native fuzz jobs for macOS and windows (take 2) 160799d9135 test: refactor: introduce `create_ephemeral_dust_package` helper 61e18dec306 doc: ephemeral policy: add missing closing double quote 32fc59796f7 rpc: Allow single transaction through submitpackage 3305972f7bf refactor: Fix remaining clang-tidy performance-unnecessary-copy-initialization errors 11f3bc229cc refactor: Reserve vectors in fuzz tests 152fefe7a22 refactor: Preallocate PrevectorFillVector(In)Direct without vector resize a774c7a339c refactor: Fix remaining clang-tidy performance-inefficient-vector errors f7144b24be0 Merge bitcoin/bitcoin#31279: policy: ephemeral dust followups c288c790cd9 interpreter: Use the same type for SignatureHash in the definition b031b7910d6 [ci] Split out native fuzz jobs for macOS and windows 6f4128e3a83 cmake, qt: Use absolute paths for includes in MOC-generated files ab5c63edcce cmake: Build `secp256k1` only when required 76a3a540a4c cmake: Ensure script correctness when no targets are specified e8f50c5debe guix: swap moreutils for just sponge 01a7298818d build: Avoid using the `-ffile-prefix-map` compiler option 2638fdb4f93 Merge bitcoin/bitcoin#31338: test: Deduplicate assert_mempool_contents() 73db95c65c1 kernel: Make bitcoin-chainstate's block validation mirror submitblock's bb53ce9bdae tests: Add functional test for submitting a previously pruned block 1f7fc738255 rpc: Remove submitblock duplicate pre-check e62a8abd7df rpc: Remove submitblock invalid-duplicate precheck 36dbebafb9b rpc: Remove submitblock coinbase pre-check 17834bd1976 Merge bitcoin/bitcoin#31333: fuzz: Implement G_TEST_GET_FULL_NAME cf577227888 Merge bitcoin/bitcoin#31335: macOS: swap docs & CI from pkg-config to pkgconf fe3457ccfff ci: note that we should install pkgconf in future a0eafc10f94 functional test: Deduplicate assert_mempool_contents() 8d203480b33 doc: migrate from pkg-config to pkgconf in macOS build docs 466e4df3fb8 assert_mempool_contents: assert not duplicates expected ea5db2f2692 functional: only generate required blocks for test d033acb6083 fuzz: package_eval: let fuzzer run out input in main tx creation loop ba35a570c5d CheckEphemeralSpends: return boolean, and set child state and txid outparams cf0cee1617c func: add note about lack of 1P1C propagation in tree submitpackage 84242903043 unit test: ephemeral_tests is using a dust relay rate, not minrelay d9cfa5fc4eb CheckEphemeralSpends: no need to iterate inputs if no parent dust 87b26e3dc07 func: rename test_free_relay to test_no_minrelay_fee e5709a4a41e func: slight elaboration on submitpackage restriction 08e969bd107 RPC: only enforce dust rules on priority when standardness active ca050d12e76 unit test: adapt to changing MAX_DUST_OUTPUTS_PER_TX 7c3490169c9 fuzz: package_eval: move last_tx inside txn ctor 445eaed182a fuzz: use optional status instead of should_rbf_eph_spend 4dfdf615b9d fuzz: remove unused TransactionsDelta validation interface 09ce926e4a1 func: cleanup reorg test comment 768a0c1889e func: cleanup test_dustrelay comments bedca1cb663 fuzz: Directly place transactions in vector c041ad6eccb fuzz: explain package eval coin tracking better bc0d98ea612 fuzz: remove dangling reference to GetEntry 15b6cbf07f5 unit test: make dust index less magical 5fbcfd12b8f unit test: assert txid returned on CheckEphemeralSpends failures ef94d84b4e4 bench: remove unnecessary CMTxn constructors c5c10fd317c ephemeral policy doxygen cleanup dd9044b8d46 ephemeral policy: IWYU c6859ce2de7 Move+rename GetDustIndexes -> GetDust 22ef95dbe3e Merge bitcoin/bitcoin#31288: Add destroy to BlockTemplate schema 92d3d691f09 fuzz: Implement G_TEST_GET_FULL_NAME f34fe0806a0 Merge bitcoin/bitcoin#31122: cluster mempool: Implement changeset interface for mempool b2d952c0f5b Merge bitcoin/bitcoin#31331: doc: add copyright header to p2p_headers_presync 7d3703dec3d doc: add copyright header to p2p_headers_presync 116b8c55736 Merge bitcoin/bitcoin#31213: fuzz: Fix difficulty target generation in `p2p_headers_presync` 15c1f47a005 Merge bitcoin/bitcoin#31327: doc: Correct PR Review Club frequency from weekly to monthly 1209a1082c8 Merge bitcoin/bitcoin#31315: build: Enable -Wbidi-chars=any ab22726def9 Merge bitcoin/bitcoin#31276: guix: scope pkg-config to Linux only 637f437a164 doc: remove PR Review Club frequency e1223099584 Merge bitcoin/bitcoin#31317: test: Revert to random path element 2666d83da59 Merge bitcoin/bitcoin#30893: test: Introduce ensure_for helper faaaf59f71e test: Make g_rng_temp_path rand, not dependent on SeedRandomForTest 746f93b4f0f Merge bitcoin/bitcoin#31307: build: Temporarily disable compiling `fuzz/utxo_snapshot.cpp` with MSVC 25fe087de59 rpc: move-only: move ScriptPubKeyDoc to utils fa80b08fef0 test: Revert to random path element 8f85d36d68a refactor: Clamp worker threads in ChainstateManager constructor fa7857ccda5 build: Enable -Wbidi-chars=any b2d53610028 build: Temporarily disable compiling `fuzz/utxo_snapshot.cpp` with MSVC 9aa50152c1c Add destroy to BlockTemplate schema ccc2d3abcd3 Merge bitcoin/bitcoin#31287: refactor: Avoid std::string format strings 62016b32300 Use std::ranges for ephemeral policy checks 3ed930a1f41 Have HasDust and PreCheckValidEphemeralTx take CTransaction 04a614bf9a7 Rename CheckValidEphemeralTx to PreCheckEphemeralTx 85bcfeea235 Merge bitcoin/bitcoin#30666: validation: fix m_best_header tracking and BLOCK_FAILED_CHILD assignment 2257c6d68fa Merge bitcoin/bitcoin#30487: ci: skip Github CI on branch pushes for forks 380e1f44e8e Merge bitcoin/bitcoin#30349: benchmark: Improve SipHash_32b accuracy to avoid potential optimization issues 1a8f51e7453 Merge bitcoin/bitcoin#28843: [refactor] Cleanup BlockAssembler mempool usage 2d944e982c4 Merge bitcoin/bitcoin#31285: guix: remove `util-linux` bcd82b13f46 Remove pkgconfig from toolchain file 319a4e82614 depends: drop sqlite pkgconfig file fa1177e3d7c refactor: Avoid std::string format strings a8fe1fd38bf depends: better cleanup after fontconfig 17e79c92607 depends: fully remove libtool archives from Qt build 8ca85651c83 guix: move pkg-config to Linux builds e3e648cf410 depends: drop pkg-config option from Qt build 0d185bd99f9 doc: update depends doc to prefer .cmake outputs e546b4e1a0c Merge bitcoin/bitcoin#31225: doc: Fix grammatical errors in multisig-tutorial.md f44e39c9d0d Merge bitcoin/bitcoin#31174: tinyformat: Add compile-time checking for literal format strings 299e2220e95 gen-manpages: implement --skip-missing-binaries 5736d1ddacc tracing: pass if replaced by tx/pkg to tracepoint a4ec07f1944 doc: add comments for CTxMemPool::ChangeSet 83f814b1d11 Remove m_all_conflicts from SubPackageState d3c8e7dfb63 Ensure that we don't add duplicate transactions in rbf fuzz tests d7dc9fd2f7b Move CalculateChunksForRBF() to the mempool changeset 284a1d33f1d Move prioritisation into changeset 446b08b599b Don't distinguish between direct conflicts and all conflicts when doing cluster-size-2-rbf checks b53041021ab Duplicate transactions are not permitted within a changeset b447416fddc Public mempool removal methods Assume() no changeset is outstanding 2b30f4d36c8 Make RemoveStaged() private 18829194ca6 Enforce that there is only one changeset at a time 7fb62f7db60 Apply mempool changeset transactions directly into the mempool 34b6c5833d1 Clean up FinalizeSubpackage to avoid workspace-specific information 57983b8add7 Move LimitMempoolSize to take place outside FinalizeSubpackage 01e145b9758 Move changeset from workspace to subpackage 802214c0832 Introduce mempool changesets 87d92fa3401 test: Add unit test coverage of package rbf + prioritisetransaction 15d982f91e6 Add package hash to package-rbf log message fa5e7064597 ci: Skip broken Wine64 tests by default 4d668549825 ci: remove util-linux from centos CI cdf34be7c94 guix: remove util-linux cbf1a47d606 CheckEphemeralSpends: only compute txid of tx when needed 111465d72dd test: Remove unused attempts parameter from wait_until 5468a23eb9a test: Add check_interval parameter to wait_until 16c87d91fd4 test: Introduce ensure_for helper a6ca8f32439 fuzz: Fix difficulty target generation in p2p_headers_presync 8610bcef9d0 ci: skip Github CI on branch pushes for forks 409d0d62937 test: enable running individual independent functional test methods ee1128ead84 doc: update stack-clash-protection comment re mingw-w64 bf47448f152 test: drop check for Windows < 10 35b898c47f8 release: target Windows 10 or later 398754e70bc depends: target Windows 10 when building for mingw-w64 ac286e0d1bd doc: Fix grammatical errors in multisig-tutorial.md fa327c77e34 util: Add ConsumeArithUInt256InRange fuzzing helper 42066f45ff5 Refactor SipHash_32b benchmark to improve accuracy and avoid optimization issues fe39acf88ff tinyformat: Add compile-time checking for literal format strings 184f34f2d0f util: Support dynamic width & precision in ConstevalFormatString 192dac1d337 [refactor] Cleanup BlockAssembler mempool usage 0bd53d913c1 test: add test for getchaintips behavior with invalid chains ccd98ea4c88 test: cleanup rpc_getchaintips.py f5149ddb9b7 validation: mark blocks building on an invalid block as BLOCK_FAILED_CHILD 783cb7337f7 validation: call RecalculateBestHeader in InvalidChainFound 9275e9689a4 rpc: call RecalculateBestHeader as part of reconsiderblock a51e91783aa validation: add RecalculateBestHeader() function REVERT: 403c20980ec kernel: Add pure kernel bitcoin-chainstate REVERT: 2e1b262eff6 kernel: Add functions to get the block hash from a block REVERT: f0b56a6052a kernel: Add block index utility functions to C header REVERT: 9393cf98179 kernel: Add function to read block undo data from disk to C header REVERT: e2ccfba0e3e kernel: Add functions to read block from disk to C header REVERT: 0559575861c kernel: Add function for copying block data to C header REVERT: 7b02e52237e kernel: Add functions for the block validation state to C header REVERT: fdebacca7ad kernel: Add validation interface to C header REVERT: f4ea5f49c6d kernel: Add interrupt function to C header REVERT: 5f4b436aad9 kernel: Add import blocks function to C header REVERT: c95a28fd80c kernel: Add chainstate load options for in-memory dbs in C header REVERT: d6360557ef6 kernel: Add options for reindexing in C header REVERT: a125867b9fe kernel: Add block validation to C header REVERT: b2b75a0ef73 Kernel: Add chainstate loading to kernel C header REVERT: d233003ff2a kernel: Add chainstate manager option for setting worker threads REVERT: 3610be3b138 kernel: Add chainstate manager object to C header REVERT: c194bea41f6 kernel: Add notifications context option to C header REVERT: 691d89d846b kerenl: Add chain params context option to C header REVERT: 407ca750cda kernel: Add kernel library context object REVERT: ee3c4ea92cd kernel: Add logging to kernel library C header REVERT: e6c610a7e03 kernel: Introduce initial kernel C header API git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 6090df267dfece6192b567fed6582445aa811e7f

20eec64b5e kernel: Add pure kernel bitcoin-chainstate 1522ee9596 kernel: Add functions to get the block hash from a block c543cadc90 kernel: Add block index utility functions to C header 6b5a4fed43 kernel: Add function to read block undo data from disk to C header 6b53c3ba1a kernel: Add functions to read block from disk to C header 7f9908ad99 kernel: Add function for copying block data to C header 1ab71ac371 kernel: Add functions for the block validation state to C header 508dd4db98 kernel: Add validation interface to C header a6a658e5f8 kernel: Add interrupt function to C header 253c3cd36f kernel: Add import blocks function to C header 3ed633b6c4 kernel: Add chainstate load options for in-memory dbs in C header 08ec37f8ce kernel: Add options for reindexing in C header ad0875e397 kernel: Add block validation to C header 2575018d69 Kernel: Add chainstate loading to kernel C header f5d21c94dc kernel: Add chainstate manager option for setting worker threads 783f56f0a2 kernel: Add chainstate manager object to C header 262039e409 kernel: Add notifications context option to C header dc0d406dd5 kerenl: Add chain params context option to C header b5f84de7ad kernel: Add kernel library context object dad0009c86 kernel: Add logging to kernel library C header 27e25aa941 kernel: Introduce initial kernel C header API ff873a20a7 Merge bitcoin/bitcoin#31313: refactor: Clamp worker threads in ChainstateManager constructor c9a7418a8d Merge bitcoin/bitcoin#31096: Package validation: accept packages of size 1 6f24662eb9 Merge bitcoin/bitcoin#31175: rpc: Remove submitblock pre-checks 3867d2421a Merge bitcoin/bitcoin#31112: Improve parallel script validation error debug logging 8e02b48059 Merge bitcoin/bitcoin#31284: ci: Skip broken Wine64 tests by default 492e1f0994 [validation] merge all ConnectBlock debug logging code paths b49df703f0 [validation] include all logged information in BlockValidationState 7b267c034f [validation] Add detailed txin/txout information for script error messages 146a3d5426 [validation] Make script error messages uniform for parallel/single validation 1ac1c33f3f [checkqueue] support user-defined return type through std::optional ebe4cac38b Merge bitcoin/bitcoin#30991: test: enable running independent functional test sub-tests 1927674100 Merge bitcoin/bitcoin#31387: doc: Use more precise anchor link to codesigning docs e043618d44 Merge bitcoin/bitcoin#31396: test: simple reordering to reduce run time a25b892ab1 Merge bitcoin/bitcoin#31386: doc: Use more precise anchor links to Xcode SDK extraction eb646111cd Merge bitcoin/bitcoin#31383: test: Add missing node.setmocktime(self.mocktime) to p2p_ibd_stalling.py 6cd95de2e0 Merge bitcoin/bitcoin#31395: build: Set shared linker flags in toolchain file abeebccc48 Merge bitcoin/bitcoin#31357: cmake: Improve build script correctness 4c9b13841c Merge bitcoin/bitcoin#31402: doc: correct libfuzzer-nosan preset flag da4f4fac8d Merge bitcoin/bitcoin#31361: cmake, qt: Use absolute paths for includes in MOC-generated files 16b140f225 doc: correct libfuzzer-nosan preset flag 097c66f614 Merge bitcoin/bitcoin#30039: dbwrapper: Bump LevelDB max file size to 32 MiB to avoid system slowdown from high disk cache flush rate 68daaea0e4 Merge bitcoin/bitcoin#31390: Remove `src/config` directory 14f162dc5c Merge bitcoin/bitcoin#31399: ci, macos: Install `pkgconf` Homebrew's package e2f2698395 ci, macos: Install `pkgconf` Homebrew's package b73d331937 dbwrapper: Bump max file size to 32 MiB 62f6d9e1a4 test: simple ordering optimization to reduce runtime a8e04704f9 build: Set shared linker flags in toolchain file dbc8ba12f3 Merge bitcoin/bitcoin#31371: doc, test: more ephemeral dust follow-ups 935973b315 Remove `src/config` directory 19f49c7489 doc: Use more precise anchor link to codesigning docs 8bf1b3039c doc: Use more precise anchor links to Xcode SDK extraction 7590e93bc7 Merge bitcoin/bitcoin#30986: contrib: skip missing binaries in gen-manpages b2af068825 Merge bitcoin/bitcoin#30708: rpc: add getdescriptoractivity 144f98db85 Merge bitcoin/bitcoin#31337: build: Fix coverage builds faa16ed4b9 test: Add missing node.setmocktime(self.mocktime) to p2p_ibd_stalling.py efdb49afb9 Merge bitcoin/bitcoin#31323: guix: swap `moreutils` for just `sponge` 37a5c5d836 doc: update descriptors.md for getdescriptoractivity ee3ce6a4f4 test: rpc: add no address case for getdescriptoractivity 811f76f3a5 rpc: add getdescriptoractivity ee6185372f gen-manpages: Prompt error if no binaries are found 70e20ea024 Merge bitcoin/bitcoin#31172: build: increase minimum supported Windows to 10.0 733317ba94 Merge bitcoin/bitcoin#31364: refactor: Fix remaining clang-tidy performance-unnecessary-copy-initialization errors 5a4bc5c036 Merge bitcoin/bitcoin#31305: refactor: Fix remaining clang-tidy performance-inefficient-vector errors 28fd0bc731 Merge bitcoin/bitcoin#31365: interpreter: Use the same type for SignatureHash in the definition 72ab35a6d0 Merge bitcoin/bitcoin#31221: ci: Split out native fuzz jobs for macOS and windows (take 2) 160799d913 test: refactor: introduce `create_ephemeral_dust_package` helper 61e18dec30 doc: ephemeral policy: add missing closing double quote 32fc59796f rpc: Allow single transaction through submitpackage 3305972f7b refactor: Fix remaining clang-tidy performance-unnecessary-copy-initialization errors 11f3bc229c refactor: Reserve vectors in fuzz tests 152fefe7a2 refactor: Preallocate PrevectorFillVector(In)Direct without vector resize a774c7a339 refactor: Fix remaining clang-tidy performance-inefficient-vector errors f7144b24be Merge bitcoin/bitcoin#31279: policy: ephemeral dust followups c288c790cd interpreter: Use the same type for SignatureHash in the definition b031b7910d [ci] Split out native fuzz jobs for macOS and windows 6f4128e3a8 cmake, qt: Use absolute paths for includes in MOC-generated files ab5c63edcc cmake: Build `secp256k1` only when required 76a3a540a4 cmake: Ensure script correctness when no targets are specified e8f50c5deb guix: swap moreutils for just sponge 01a7298818 build: Avoid using the `-ffile-prefix-map` compiler option 2638fdb4f9 Merge bitcoin/bitcoin#31338: test: Deduplicate assert_mempool_contents() 73db95c65c kernel: Make bitcoin-chainstate's block validation mirror submitblock's bb53ce9bda tests: Add functional test for submitting a previously pruned block 1f7fc73825 rpc: Remove submitblock duplicate pre-check e62a8abd7d rpc: Remove submitblock invalid-duplicate precheck 36dbebafb9 rpc: Remove submitblock coinbase pre-check 17834bd197 Merge bitcoin/bitcoin#31333: fuzz: Implement G_TEST_GET_FULL_NAME cf57722788 Merge bitcoin/bitcoin#31335: macOS: swap docs & CI from pkg-config to pkgconf fe3457ccff ci: note that we should install pkgconf in future a0eafc10f9 functional test: Deduplicate assert_mempool_contents() 8d203480b3 doc: migrate from pkg-config to pkgconf in macOS build docs 466e4df3fb assert_mempool_contents: assert not duplicates expected ea5db2f269 functional: only generate required blocks for test d033acb608 fuzz: package_eval: let fuzzer run out input in main tx creation loop ba35a570c5 CheckEphemeralSpends: return boolean, and set child state and txid outparams cf0cee1617 func: add note about lack of 1P1C propagation in tree submitpackage 8424290304 unit test: ephemeral_tests is using a dust relay rate, not minrelay d9cfa5fc4e CheckEphemeralSpends: no need to iterate inputs if no parent dust 87b26e3dc0 func: rename test_free_relay to test_no_minrelay_fee e5709a4a41 func: slight elaboration on submitpackage restriction 08e969bd10 RPC: only enforce dust rules on priority when standardness active ca050d12e7 unit test: adapt to changing MAX_DUST_OUTPUTS_PER_TX 7c3490169c fuzz: package_eval: move last_tx inside txn ctor 445eaed182 fuzz: use optional status instead of should_rbf_eph_spend 4dfdf615b9 fuzz: remove unused TransactionsDelta validation interface 09ce926e4a func: cleanup reorg test comment 768a0c1889 func: cleanup test_dustrelay comments bedca1cb66 fuzz: Directly place transactions in vector c041ad6ecc fuzz: explain package eval coin tracking better bc0d98ea61 fuzz: remove dangling reference to GetEntry 15b6cbf07f unit test: make dust index less magical 5fbcfd12b8 unit test: assert txid returned on CheckEphemeralSpends failures ef94d84b4e bench: remove unnecessary CMTxn constructors c5c10fd317 ephemeral policy doxygen cleanup dd9044b8d4 ephemeral policy: IWYU c6859ce2de Move+rename GetDustIndexes -> GetDust 22ef95dbe3 Merge bitcoin/bitcoin#31288: Add destroy to BlockTemplate schema 92d3d691f0 fuzz: Implement G_TEST_GET_FULL_NAME f34fe0806a Merge bitcoin/bitcoin#31122: cluster mempool: Implement changeset interface for mempool b2d952c0f5 Merge bitcoin/bitcoin#31331: doc: add copyright header to p2p_headers_presync 7d3703dec3 doc: add copyright header to p2p_headers_presync 116b8c5573 Merge bitcoin/bitcoin#31213: fuzz: Fix difficulty target generation in `p2p_headers_presync` 15c1f47a00 Merge bitcoin/bitcoin#31327: doc: Correct PR Review Club frequency from weekly to monthly 1209a1082c Merge bitcoin/bitcoin#31315: build: Enable -Wbidi-chars=any ab22726def Merge bitcoin/bitcoin#31276: guix: scope pkg-config to Linux only 637f437a16 doc: remove PR Review Club frequency e122309958 Merge bitcoin/bitcoin#31317: test: Revert to random path element 2666d83da5 Merge bitcoin/bitcoin#30893: test: Introduce ensure_for helper faaaf59f71 test: Make g_rng_temp_path rand, not dependent on SeedRandomForTest 746f93b4f0 Merge bitcoin/bitcoin#31307: build: Temporarily disable compiling `fuzz/utxo_snapshot.cpp` with MSVC 25fe087de5 rpc: move-only: move ScriptPubKeyDoc to utils fa80b08fef test: Revert to random path element 8f85d36d68 refactor: Clamp worker threads in ChainstateManager constructor fa7857ccda build: Enable -Wbidi-chars=any b2d5361002 build: Temporarily disable compiling `fuzz/utxo_snapshot.cpp` with MSVC 9aa50152c1 Add destroy to BlockTemplate schema ccc2d3abcd Merge bitcoin/bitcoin#31287: refactor: Avoid std::string format strings 62016b3230 Use std::ranges for ephemeral policy checks 3ed930a1f4 Have HasDust and PreCheckValidEphemeralTx take CTransaction 04a614bf9a Rename CheckValidEphemeralTx to PreCheckEphemeralTx 85bcfeea23 Merge bitcoin/bitcoin#30666: validation: fix m_best_header tracking and BLOCK_FAILED_CHILD assignment 2257c6d68f Merge bitcoin/bitcoin#30487: ci: skip Github CI on branch pushes for forks 380e1f44e8 Merge bitcoin/bitcoin#30349: benchmark: Improve SipHash_32b accuracy to avoid potential optimization issues 1a8f51e745 Merge bitcoin/bitcoin#28843: [refactor] Cleanup BlockAssembler mempool usage 2d944e982c Merge bitcoin/bitcoin#31285: guix: remove `util-linux` bcd82b13f4 Remove pkgconfig from toolchain file 319a4e8261 depends: drop sqlite pkgconfig file fa1177e3d7 refactor: Avoid std::string format strings a8fe1fd38b depends: better cleanup after fontconfig 17e79c9260 depends: fully remove libtool archives from Qt build 8ca85651c8 guix: move pkg-config to Linux builds e3e648cf41 depends: drop pkg-config option from Qt build 0d185bd99f doc: update depends doc to prefer .cmake outputs e546b4e1a0 Merge bitcoin/bitcoin#31225: doc: Fix grammatical errors in multisig-tutorial.md f44e39c9d0 Merge bitcoin/bitcoin#31174: tinyformat: Add compile-time checking for literal format strings 299e2220e9 gen-manpages: implement --skip-missing-binaries 5736d1ddac tracing: pass if replaced by tx/pkg to tracepoint a4ec07f194 doc: add comments for CTxMemPool::ChangeSet 83f814b1d1 Remove m_all_conflicts from SubPackageState d3c8e7dfb6 Ensure that we don't add duplicate transactions in rbf fuzz tests d7dc9fd2f7 Move CalculateChunksForRBF() to the mempool changeset 284a1d33f1 Move prioritisation into changeset 446b08b599 Don't distinguish between direct conflicts and all conflicts when doing cluster-size-2-rbf checks b53041021a Duplicate transactions are not permitted within a changeset b447416fdd Public mempool removal methods Assume() no changeset is outstanding 2b30f4d36c Make RemoveStaged() private 18829194ca Enforce that there is only one changeset at a time 7fb62f7db6 Apply mempool changeset transactions directly into the mempool 34b6c5833d Clean up FinalizeSubpackage to avoid workspace-specific information 57983b8add Move LimitMempoolSize to take place outside FinalizeSubpackage 01e145b975 Move changeset from workspace to subpackage 802214c083 Introduce mempool changesets 87d92fa340 test: Add unit test coverage of package rbf + prioritisetransaction 15d982f91e Add package hash to package-rbf log message fa5e706459 ci: Skip broken Wine64 tests by default 4d66854982 ci: remove util-linux from centos CI cdf34be7c9 guix: remove util-linux cbf1a47d60 CheckEphemeralSpends: only compute txid of tx when needed 111465d72d test: Remove unused attempts parameter from wait_until 5468a23eb9 test: Add check_interval parameter to wait_until 16c87d91fd test: Introduce ensure_for helper a6ca8f3243 fuzz: Fix difficulty target generation in p2p_headers_presync 8610bcef9d ci: skip Github CI on branch pushes for forks 409d0d6293 test: enable running individual independent functional test methods ee1128ead8 doc: update stack-clash-protection comment re mingw-w64 bf47448f15 test: drop check for Windows < 10 35b898c47f release: target Windows 10 or later 398754e70b depends: target Windows 10 when building for mingw-w64 ac286e0d1b doc: Fix grammatical errors in multisig-tutorial.md fa327c77e3 util: Add ConsumeArithUInt256InRange fuzzing helper 42066f45ff Refactor SipHash_32b benchmark to improve accuracy and avoid optimization issues fe39acf88f tinyformat: Add compile-time checking for literal format strings 184f34f2d0 util: Support dynamic width & precision in ConstevalFormatString 192dac1d33 [refactor] Cleanup BlockAssembler mempool usage 0bd53d913c test: add test for getchaintips behavior with invalid chains ccd98ea4c8 test: cleanup rpc_getchaintips.py f5149ddb9b validation: mark blocks building on an invalid block as BLOCK_FAILED_CHILD 783cb7337f validation: call RecalculateBestHeader in InvalidChainFound 9275e9689a rpc: call RecalculateBestHeader as part of reconsiderblock a51e91783a validation: add RecalculateBestHeader() function REVERT: 35f8503285 kernel: Add pure kernel bitcoin-chainstate REVERT: 84eb1f952c kernel: Add functions to get the block hash from a block REVERT: 575cb5a033 kernel: Add block index utility functions to C header REVERT: 4c433defd3 kernel: Add function to read block undo data from disk to C header REVERT: 83e48e021b kernel: Add functions to read block from disk to C header REVERT: a4381c560f kernel: Add function for copying block data to C header REVERT: d3e84ac5a6 kernel: Add functions for the block validation state to C header REVERT: deb5b4a5f5 kernel: Add validation interface to C header REVERT: f4ea5f49c6 kernel: Add interrupt function to C header REVERT: 5f4b436aad kernel: Add import blocks function to C header REVERT: c95a28fd80 kernel: Add chainstate load options for in-memory dbs in C header REVERT: d6360557ef kernel: Add options for reindexing in C header REVERT: a125867b9f kernel: Add block validation to C header REVERT: b2b75a0ef7 Kernel: Add chainstate loading to kernel C header REVERT: d233003ff2 kernel: Add chainstate manager option for setting worker threads REVERT: 3610be3b13 kernel: Add chainstate manager object to C header REVERT: c194bea41f kernel: Add notifications context option to C header REVERT: 691d89d846 kerenl: Add chain params context option to C header REVERT: 407ca750cd kernel: Add kernel library context object REVERT: ee3c4ea92c kernel: Add logging to kernel library C header REVERT: e6c610a7e0 kernel: Introduce initial kernel C header API git-subtree-dir: depend/bitcoin git-subtree-split: 20eec64b5e417cac8c68100826c0adf2152a49eb

LarryRuane · 2025-01-06T18:48:31Z

@sipa:

Concept ACK. Please update the PR title and description to reflect the new size.

If it is still possible post-merge, please update the description; it still says 128 MiB (the title has been updated), thanks.

…64-bit systems" After bitcoin/bitcoin#30039, the number of ldb files created is 16 times smaller. 1000 files with the new default of 32MB is 32GB of database. If we need more, we can increase the default file size again. This patch seems unnecessary now. This reverts commit 92ae82c.

fanquake · 2025-05-08T12:35:55Z

Anyone that reviewed here, might be interested in bitcoin-core/leveldb-subtree#52.

Github-Pull: bitcoin#30039 Rebased-From: c283a57

laanwj added the UTXO Db and Indexes label May 4, 2024

andrewtoth reviewed May 6, 2024

View reviewed changes

luke-jr mentioned this pull request May 8, 2024

Add option dbfilesize to control LevelDB target ("max") file size #30059

Open

mzumsande reviewed May 8, 2024

View reviewed changes

DrahtBot mentioned this pull request Jun 27, 2024

kernel, logging: Pass Logger instances to kernel objects #30342

Draft

davidgumberg mentioned this pull request Oct 1, 2024

Don't zero-after-free DataStream: Faster IBD on some configurations #30987

Draft

l0rinc reviewed Oct 30, 2024

View reviewed changes

src/dbwrapper.cpp Show resolved Hide resolved

l0rinc mentioned this pull request Nov 18, 2024

max_file_size = 16 << 20 bitcoin-dev-tools/benchcoin#45

Closed

willcl-ark approved these changes Nov 30, 2024

View reviewed changes

DrahtBot requested a review from sipa November 30, 2024 20:24

tdb3 approved these changes Nov 30, 2024

View reviewed changes

laanwj approved these changes Nov 30, 2024

View reviewed changes

fanquake merged commit 097c66f into bitcoin:master Dec 2, 2024
18 checks passed

This was referenced Dec 4, 2024

validation: fetch block inputs on parallel threads 10% faster IBD #31132

Open

validation: write chainstate to disk every hour #30611

Merged

l0rinc mentioned this pull request Dec 5, 2024

[IBD] multi-byte block obfuscation #31144

Merged

This was referenced Jan 6, 2025

UX: enabling lightwalletd index (indices) makes initial sync (IBD) very slow zcash/zcash#6904

Closed

dbwrapper: Bump max file size to 32 MiB (lightwalletd sync performance) zcash/zcash#6997

Merged

andrewtoth mentioned this pull request Feb 12, 2025

Add -pruneduringinit option to temporarily use another prune target during IBD #31845

Open

l0rinc mentioned this pull request Mar 4, 2025

Measure used memory bitcoin-dev-tools/benchcoin#143

Open

This was referenced Mar 12, 2025

[IBD] Tracking PR for speeding up Initial Block Download #32043

Draft

[IBD] flush UTXO set in batches proportional to dbcache size #31645

Open

johnny9 mentioned this pull request Mar 23, 2025

Socratic Seminar 39 TriangleBitDevs/TriangleBitDevs.github.io#36

Closed

laanwj mentioned this pull request May 8, 2025

[RFC] dbwrapper: Set global leveldb mmap limit #32447

Closed

laanwj mentioned this pull request May 8, 2025

Revert "Increase maximum read-only mmap()s used from 1000 to 4096 on 64-bit systems" bitcoin-core/leveldb-subtree#52

Open

l0rinc mentioned this pull request May 31, 2025

bitcoind 29.0 much slower than 28.0 on my system: cause found #32455

Closed

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Jun 6, 2025

Add option dbfilesize to control LevelDB target ("max") file size

8d25f91

Github-Pull: bitcoin#30039 Rebased-From: c283a57

l0rinc mentioned this pull request Jul 2, 2025

bitcoind shouldn't fail to progress with synchronization: endless [leveldb] Generated table ... logs #31882

Open

1 task

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Jul 29, 2025

Add option dbfilesize to control LevelDB target ("max") file size

9d7ef52

Github-Pull: bitcoin#30039 Rebased-From: c283a57

dbwrapper: Bump LevelDB max file size to 32 MiB to avoid system slowdown from high disk cache flush rate #30039

dbwrapper: Bump LevelDB max file size to 32 MiB to avoid system slowdown from high disk cache flush rate #30039

Uh oh!

Conversation

maciejsszmigiero commented May 4, 2024

Uh oh!

DrahtBot commented May 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage & Benchmarks

Reviews

Conflicts

Uh oh!

sipa commented May 4, 2024

Uh oh!

laanwj commented May 4, 2024

Uh oh!

willcl-ark commented May 4, 2024

Uh oh!

maciejsszmigiero commented May 4, 2024

Uh oh!

laanwj commented May 4, 2024

Uh oh!

tdb3 commented May 5, 2024

Uh oh!

andrewtoth commented May 5, 2024

Uh oh!

maciejsszmigiero commented May 5, 2024

Uh oh!

andrewtoth May 6, 2024

Choose a reason for hiding this comment

Uh oh!

laanwj May 6, 2024

Choose a reason for hiding this comment

Uh oh!

maciejsszmigiero May 7, 2024

Choose a reason for hiding this comment

Uh oh!

andrewtoth commented May 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented May 6, 2024

Uh oh!

luke-jr commented May 7, 2024

Uh oh!

willcl-ark commented May 7, 2024

Uh oh!

andrewtoth commented May 7, 2024

Uh oh!

maciejsszmigiero commented May 7, 2024

Uh oh!

mzumsande left a comment

Choose a reason for hiding this comment

Uh oh!

sipa commented May 17, 2024

Uh oh!

l0rinc commented Oct 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidgumberg commented Oct 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

l0rinc commented Oct 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

l0rinc commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maciejsszmigiero commented Nov 9, 2024

Uh oh!

willcl-ark commented Nov 22, 2024

Uh oh!

willcl-ark left a comment

Choose a reason for hiding this comment

Uh oh!

tdb3 left a comment

Choose a reason for hiding this comment

Uh oh!

laanwj left a comment

DrahtBot commented May 4, 2024 •

edited

Loading

andrewtoth commented May 6, 2024 •

edited

Loading

l0rinc commented Oct 2, 2024 •

edited

Loading

davidgumberg commented Oct 4, 2024 •

edited

Loading

l0rinc commented Oct 7, 2024 •

edited

Loading

l0rinc commented Nov 6, 2024 •

edited

Loading