streams: cache file position within AutoFile #30884

sipa · 2024-09-12T15:57:22Z

Instead of relying on frequent ftell calls (which appear to cause a significant slowdown on some systems) in XOR-enabled AutoFiles, cache the file position within AutoFile itself.

DrahtBot · 2024-09-12T15:57:25Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	theStack, achow101, davidgumberg

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#29641 (scripted-diff: Use LogInfo over LogPrintf by maflcko)
#29307 (util: explicitly close all AutoFiles that have been written by vasild)
#28792 (Embed default ASMap as binary dump header file by fjahr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

davidgumberg · 2024-09-12T20:45:57Z

Nice! I am still running benchmarks, but early results indicate that this addresses #30833:

In my current run, up to block 240,000 v28.0rc1 bitcoind spent 967.67s writing undo data and your branch cherry picked onto the v28.x branch spents 1640.60s writing undo data, and like andrewtoth mentioned a lot of the speed regression as measured by wall clocks comes from AcceptBlock which doesn't have any bench logs.

I'll update with the wall clock times once my bench runs finish.

I think that there are also some calls to std::rewind:

bitcoin/src/test/streams_tests.cpp

Line 264 in cf0120f

std::rewind(file.Get());

and std::fgetc:

bitcoin/src/node/utxo_snapshot.cpp

Line 76 in cf0120f

if (std::fgetc(afile.Get()) != EOF) {

that are problematic if we are caching file position.

Also, if it seems sensible to mitigate the risk of someone modifying the file position in the future without changing m_position, we could drop AutoFile::Get() entirely, e.g.: davidgumberg@d238c46

sipa · 2024-09-12T21:43:31Z

@davidgumberg Good points. I've made minimal changes in this PR to make sure no position-effecting operations remain, as we're past the feature freeze. Extending the AutoFile interface so that AutoFile::Get is no longer needed can be done as a follow-up.

vostrnad · 2024-09-12T23:58:24Z

I'm happy to report that a quick benchmark of 4cfff4e cherry-picked onto v28.0rc1 shows that the slowdown is almost or entirely gone. I'll do a more thorough benchmark later once the code is ACKed.

davidgumberg · 2024-09-13T02:48:32Z

src/streams.cpp

+    } else {
+        int64_t r{std::ftell(m_file)};
+        if (r < 0) {
+            throw std::ios_base::failure("AutoFile::seek: ftell failed");
+        }
+        m_position = r;


Only commenting because I struggled with this for a bit:

This makes sense since the only present possibility for this else branch is that origin == SEEK_END, in which case the only way for fseek to have returned zero is if offset is a negative number indicating how far from the end of the file to seek to, and m_position = length_of_file + negative_offset

But, in order to get the length of the file you need to fseek(m_file, 0, SEEK_END) and ftell(m_file).

So, just ftell here is the best way to get the new m_position.

davidgumberg · 2024-09-13T04:02:07Z

src/streams.cpp

-        std::size_t ret{std::fread(dst.data(), 1, dst.size(), m_file)};
-        util::Xor(dst.subspan(0, ret), m_xor, init_pos);
-        return ret;
+    size_t ret = std::fread(dst.data(), 1, dst.size(), m_file);


Outside of the scope of this PR: I feel like this should probably handle a failed fread explicitly, even though the subspan call below means XOR won't do anything bad here, and when this returns to AutoFile::read() or BufferedFile::fill() (it's only two callers) the error gets handled. It seems kind of footgun-like to let this potentially dangerous file pointer get passed around and I'm not sure it's worth BufferedFile being able to print it's own name when throwing an exception.

Handling i/o errors (eg partial reads/writes can be considered fatal for disk i/o) is always important, but yes even more-so now that the file pointer is cached and might get out of sync with the world.

src/streams.cpp

davidgumberg · 2024-09-13T05:13:19Z

src/streams.cpp

+{
+    if (!IsNull()) {
+        auto pos{std::ftell(m_file)};
+        if (pos >= 0) m_position = pos;


nit: Handle ftell error

Suggested change

if (pos >= 0) m_position = pos;

if (pos < 0) {

throw std::ios_base::failure("AutoFile::seek: ftell failed");

}

m_position = pos;

I tried that and it caused test failures. My reasoning is that the approach here is fine, because a failure to tell what position a FILE* is at means that further operations will fail too, anyway.

And there are also files for which fseek/ftell just don't work (e.g. pipes/fifos, and /proc files). As long as no xoring is in use, this should be fine.

EDIT: see update below; I've made m_position an optional so it can represent "unknown position" instead.

That makes sense, I didn't consider pipes/fifos.

I agree on this approach given that a bad m_position value is only an issue when XOR'ing, and your branch has checks for an unknown m_position before any XOR operations. And in general, it seems to me, ftell's are followed by file operations that fail if the ftell fails for any reason other than the file being a pipe/fifo (ESPIPE).

I think ideally errno is checked here for values other than ESPIPE, but the current approach is good.

davidgumberg · 2024-09-13T06:35:29Z

src/streams.cpp

@@ -60,6 +65,7 @@ void AutoFile::ignore(size_t nSize)
            throw std::ios_base::failure(feof() ? "AutoFile::ignore: end of file" : "AutoFile::ignore: fread failed");
        }
        nSize -= nNow;
+        m_position += nNow;
    }
 }


Out of scope but I found while reviewing: Is there any reason why AutoFile::ignore(nSize) couldn't be removed and AutoFile::seek(nSize, SEEK_CURR) used instead?

Maybe it would be okay to just leave ignore as an alias for seek if the semantics are sufficiently desirable.

I believe that when ignore is used, it's usually for small amounts of data, which are likely already cached.

Is there any reason why AutoFile::ignore(nSize) couldn't be removed and AutoFile::seek(nSize, SEEK_CURR) used instead?

Keep in mind not all streams support seeking, so it would at the least be a conditional optimization. Agree it's not worth it as long as the amounts skipped are small (generally smaller than the page size).

davidgumberg · 2024-09-13T08:37:12Z

crACK 4cfff4e

Awesome! My testing (n=2) indicates that this branch solves the Windows IBD regression reported in #30833.

In my bench runs your branch cherry picked on 28.x took 3% more wall clock time than v27.1 (3% is smaller than the fluctuations I observed during runs on the same branch), but v28rc1 took 112% more time to reach block 300k than 27.1.

I left a nit and some out-of-scope observations about some of the areas touched here most of which are best addressed in follow up PR's. I tried to be as thorough as possible and could not think of a way for m_position to fall out of sync with the OS's file pointer in the present branch. I think a future PR should definitely remove the AutoFile::Get() method to avoid users of the AutoFile interface from interacting with the raw FILE* and reintroducing calls that could desynchronize it.

Given the severity of a potential issue, it might be nice to add some fuzz/test checks that make sure that m_position desync doesn't happen.

Benchmarks

I ran the same test (n=2) on three versions of Bitcoin Core: the v27.1 tag, the v28.0rc1 tag, and the 28.x branch with 8a7d8f7 ¹ cherry picked, all built the same way using mingw on a separate machine and without debug symbols stripped.

Build commands

git clean -dfx
cd depends && make HOST=x86_64-w64-mingw32 -j $(nproc) && cd ../
./autogen.sh && CONFIG_SITE=$PWD/depends/x86_64-w64-mingw32/share/config.site ./configure --prefix=/
make -j $(nproc)

I ran all three tests on a Thinkpad T470 with an i5-6300U running Windows 10 Enterprise LTSC Build 17763. It has an SSD that my butt-dyno reports is pretty slow, which I suspect is important to reproduce #30833.

I used the following command for all six runs, clearing the datadir between each run:

.\bitcoind.exe  -stopatheight=300000 -dbcache=450 -debug=bench -connect=fastlocalnode:8333 -datadir=C:\tmpdatadir -printtoconsole=0

Mean times to 300k (2 runs):

	Mean time
v27.1	0h 52min 21sec
v28rc1	1h 50min 47sec
28.x + #30833	0hr 53min 52sec

Here is the data from my individual runs, in your branch, write undo time are even smaller than v27.1

	Wall-clock to 300k blocks	Write undo time	BlockConnect Total
27.1 Run 1	0hr 53min 52sec	1,595.79s	2,164.82s
27.1 Run 2	0hr 50min 49sec	1,570.92s	2,146.43s
28rc1 Run 1	1hr 49min 27sec	2,707.95s	3,231.60s
28rc1 Run 2	1hr 52min 07sec	2,547.99s	3,091.75s
28.x + #30884 run 1	0hr 52min 34sec	1,138.34s	1,714.37s
28.x + #30884 run 2	0hr 55min 09sec	968.53s	1,535.17s

Out of the scope of this PR: One thing I find notable about the above data is that it seems that 28.x block connection times are down pretty significantly from 27.1, but wall clock IBD times are similar, indicating that there is a regression in some place other than ConnectBlock() at least on this machine with this setup, I don't think it's significant enough to block this release candidate, but I will investigate further.

Why the regression happened

I suspect that as suggested on IRC, on some systems, an ftell call that follows an fwrite triggers a flush. Apparently, one of the naive ways to implement ftell is as fseek(fp, 0, SEEK_CUR) but returning the fp offset. And an fseek after an fwrite results in a flush.² This is exactly the way glibc implemented this until in 2012 a patch was introduced which fixed this exact performance issue of extraneous flushes happening in loops where you ftell and fwrite many times by decoupling ftell calls from fseek:

glibc mailing list: [PATCH][BZ #5298] Don't flush write buffer for ftell

Hi,

The current implementation of ftell is basically equivalent to
fseek(fp, 0, SEEK_CUR). While this is not incorrect, it results in
inheritance of limitations of fseek, which is summarized in the
following comment in the source:

  /* Flush unwritten characters.
     (This may do an unneeded write if we seek within the buffer.
     But to be able to switch to reading, we would need to set
     egptr to ptr.  That can't be done in the current design,
     which assumes file_ptr() is eGptr.  Anyway, since we probably
     end up flushing when we close(), it doesn't make much difference.)
     FIXME: simulate mem-papped files. */

This is not needed for ftell since it does not need to set or
modify buffer state, so this flush can be avoided. Attached patch
computes current position for ftell (within the file_seekoff functions
as a special case) without flushing the buffers when in write mode. I
have used a modified version of the sample program in the bz (appended
to this email) to check the improvement in performance in each call and
the average reads as below on my Fedora 16 x86_64 core i5 laptop with
4GB RAM:

Without patch:
Total time: 9174470.000000 ns. AVG 1819.609282 ns per call

With patch:
Total time: 1047375.000000 ns. AVG 207.730067 ns per call

I have verified that the change does not cause any regressions in the
testsuite.

Regards,
Siddhesh

Here is the relevant bugzilla issue: https://sourceware.org/bugzilla/show_bug.cgi?id=5298
and a mirror of the commit: bminor/glibc@adb26fa

I feel it's likely that Windows libc either made the same choice, or some other design constraint forces windows to flush on ftell.

What follows is vague and mistaken:

The reason ftell should act like fseek at all is because in the ballet that the 6 internal streambuffer pointers (eback, gptr, egptr, pbase, pptr, epptr) do as reads and writes happen, read pointer (gptr) != write pointer(pptr) != the file position offset, and especially after a write, pptr advances past egptr (which it seems in the case of file i/o is usually == gptr??) . So, the naive solution to this problem is to reuse fseek which, according to some C standards documents (https://pubs.opengroup.org/onlinepubs/007904975/functions/fseek.html),

The following is guaranteed as an effect of fseek():

If the stream is writable and buffered data had not been written to the underlying file, fseek() shall cause the unwritten data to be written to the file and shall mark the st_ctime and st_mtime fields of the file for update.

So fseek brings all into happy sync, and now egptr == gptr == pptr.

But this is not necessary for ftell which does not need any updates to the streambuf's internal state, and can basically just use do some pointer arithmetic from pptr, egptr, and the current offset to find the new offset without needing to flush anything to disk.

Information about the pointer arithmetic in this mailing list post: https://sourceware.org/legacy-ml/libc-alpha/2012-09/msg00631.html
and relevant source code for calculating the offset for ftell: https://github.com/bminor/glibc/blob/adb26faefe47b7d34c941cbfc193ca7a5fde8e3f/libio/fileops.c#L1016-L1023
and relevant source code for the flush (_IO_OVERFLOW) that fseek` does to every buffer that
arrives written and unflushed: https://github.com/bminor/glibc/blob/c9154cad66aa0b11ede62cc9190d3485c5ef6941/libio/genops.c#L189-L209

Not the most current branch but the follow-up rebase https://github.com/bitcoin/bitcoin/pull/30884/commits/4cfff4e58c6d806e4bc5a12386f84ff207c83419 makes trivial changes that should not affect these benchmarks. ↩
From IEEE Std 1003.1, 2004 Edition: If the stream is writable and buffered data had not been written to the underlying file, fseek() shall cause the unwritten data to be written to the file and shall mark the st_ctime and st_mtime fields of the file for update. ↩

sipa · 2024-09-13T11:33:18Z

I made a significant change here, making m_position an std::optional<int64_t>, which is std::nullopt when the position is unknown (due to ftell failing, and no successful fseek afterwards).

The position being unknown will permit normal operation, but throws an error on read/write when a xor key is set, or when invoking tell.

laanwj · 2024-09-13T13:19:47Z

I made a significant change here, making m_position an std::optional<int64_t>, which is std::nullopt when the position is unknown (due to ftell failing, and no successful fseek afterwards).

i first couldn't think of a valid scenario where ftell would fail but we'd want to continue, but i guess this maks sense when reading a FIFO or device, when there isn't really an offset.

achow101 · 2024-09-13T16:13:50Z

ACK e624a9b

Extending the AutoFile interface so that AutoFile::Get is no longer needed can be done as a follow-up.

It does seem a bit fragile to still have AutoFile::Get. I would okay with backporting additional commits that remove it if they are not too complicated.

Co-Authored-By: David Gumberg <davidzgumberg@gmail.com>

sipa · 2024-09-14T02:02:33Z

@achow101 I've added a commit that gets rid of AutoFile::Get entirely (based on @davidgumberg's d238c46).

vostrnad · 2024-09-15T17:52:41Z

I've benchmarked commit e624a9b, both in this branch and cherry-picked onto a few selected commits (fresh sync to block 400,000 from a local node, using default settings, all binaries self-built using Mingw-w64, around 10 runs for each version).

Summary: some slowdown due to #28052 definitely remains, but it's under 5%.

version	result	note
`fa7f7ac`	2112 ± 17 seconds	First commit of #28052, and last commit before the regression.
`fa7f7ac` + `e624a9b`	2116 ± 19 seconds	As above, but with this PR cherry-picked to verify it doesn't on its own introduce any slowdown (it doesn't).
`fa895c7` + `e624a9b`	2209 ± 17 seconds	Top commit of #28052 with this PR cherry-picked. This brings the slowdown introduced in #28052 to around 4-5%.
`e624a9b`	2144 ± 12 seconds	This PR, not cherry-picked onto anything.

theStack

Code-review ACK a240e15

Didn't run any benchmarks to compare with master.

theStack · 2024-09-15T23:08:45Z

src/node/utxo_snapshot.cpp

    } else if (std::ferror(afile.Get())) {
        LogPrintf("[snapshot] warning: i/o error reading %s\n", read_from_str);


Possible follow-up material: now that that the read operation above (std::fgetc) has been replaced by only fseek/ftell calls on the file object, I think this condition can't be true anymore and hence this else-branch could be removed.

Done in #30927.

achow101 · 2024-09-16T16:01:23Z

ACK a240e15

davidgumberg · 2024-09-16T21:45:14Z

src/streams.cpp

+
+bool AutoFile::Truncate(unsigned size)
+{
+    return ::TruncateFile(m_file, size);


Note: It is okay to do this without worrying about m_position because ftruncate does not modify the position of the file seek pointer. (https://pubs.opengroup.org/onlinepubs/007904975/functions/ftruncate.html)

davidgumberg · 2024-09-16T22:05:18Z

untested reACK a240e15
Changes since 4cfff4e should not change any behavior, they only make the AutoFile interfaces safer by:

Ensuring we only perform XOR operations or AutoFile::tell() if the file has a sensible seek pointer; e.g. if you try to ftell a pipe: ftell(stdout) -1 will be returned, and errno will be set to ESPIPE, even though it might still be valid to read and write from the "file".
Removing the AutoFile::Get() interface which allows AutoFile users to access the raw FILE pointer and potentially modify the OS file position without modifying m_position.

Github-Pull: bitcoin#30884 Rebased-From: e624a9b

Co-Authored-By: David Gumberg <davidzgumberg@gmail.com> Github-Pull: bitcoin#30884 Rebased-From: a240e15

achow101 · 2024-09-17T03:10:45Z

Backported in #30827

laanwj · 2024-09-17T09:55:14Z

Code review ACK a240e15

06a7df7 doc: Generate manpages (Ava Chow) 5315886 build: Bump to 28.0rc2 (Ava Chow) ff95cb3 streams: remove AutoFile::Get() entirely (Pieter Wuille) 8229e98 streams: cache file position within AutoFile (Pieter Wuille) 1b853fd qt: Translations update (Hennadii Stepanov) 674dded gui: fix crash when closing wallet (furszy) d39262e test: Wait for local services to update in feature_assumeutxo (Fabian Jahr) b329ed7 test: add coverage for assumeUTXO honest peers disconnection (furszy) c6b5db1 assumeUTXO: fix peers disconnection during sync (furszy) 598415b test: Work around boost compilation error (MarcoFalke) Pull request description: * #30834 * #30807 * #30880 * bitcoin-core/gui#835 * #30899 * #30884 ACKs for top commit: stickies-v: ACK 06a7df7 hebasto: ACK 06a7df7, I've backported the listed PRs locally. The only merge conflict I faced was in #30807. It was trivial to resolve. Tree-SHA512: 779d734b50fdce379a20865ba30c969def028963ba51da0f497ddf1b5375e1f6166365295f226c1a07bab8be0c1aa0a6a3296fc6acd9fcf17bcc4874aac980a6

maflcko

review ACK a240e15 🍽

Show signature

Signature:

untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: review ACK a240e150e837b5a95ed19765a2e8b7c5b6013f35 🍽
E+QumBxAsjs6onMxIQw5fqCgJWho8UUHBtPokAtkGM+wtysIawMw5OCTX9k7LQyI6+s2neQYq1Xh/+7Ifg6bCw==

maflcko · 2024-09-19T09:44:50Z

src/node/blockstorage.cpp

    if (fileOutPos < 0) {
        LogError("%s: ftell failed\n", __func__);
        return false;


Isn't this dead code now? Previously it was required, because std::ftell could fail and return -1L.

However, now that AutoFile::tell() is used, which does not return an error value (but throws on error), this code can never execute.

A negative value here could only happen if seeking prior to the start of a file was possible, which I don't think it is. Also, the AutoFile constructor already assumes this isn't possible.

In any case, the prior line writes to the file, which isn't possible/supported for negative absolute positions, so this must be dead code in this function either way.

Done in #30927.

maflcko · 2024-09-19T09:45:04Z

src/node/blockstorage.cpp

    if (fileOutPos < 0) {
        LogError("%s: ftell failed\n", __func__);
        return false;


Done in #30927.

…d code caac06f streams: reorder/document functions (Pieter Wuille) 67a3d59 streams: remove unused code (Pieter Wuille) Pull request description: This is a follow-up to #30884. Remove a number of dead code paths, and improve the code organization and documentation, in `AutoFile`. ACKs for top commit: maflcko: re-ACK caac06f theStack: Code-review ACK caac06f l0rinc: ACK caac06f tdb3: CR ACK caac06f Tree-SHA512: 297791f093e0142730f815c11dd3466b98f7e7edea86094a815dae989ef40d8056db10e0fed6e575d530903c18e80c08d36d3f1e6b828f2d955528f365b22008

…47757ea3b 1047757ea3b kernel: Add pure kernel bitcoin-chainstate c568fdf75fd kernel: Add block index utility functions to C header 0f1da1dcba5 kernel: Add function to read block undo data from disk to C header 45af559c9f6 kernel: Add functions to read block from disk to C header 2a7f8a8240c kernel: Add function for copying block data to C header b19f5336c03 kernel: Add functions for the block validation state to C header 9c0ffa913f4 kernel: Add validation interface to C header a93318c6152 kernel: Add interrupt function to C header 51053f33720 kernel: Add import blocks function to C header 6b0ada2af42 kernel: Add chainstate load options for in-memory dbs in C header 34427bfa9c7 kernel: Add options for reindexing in C header ca57311c969 kernel: Add block validation to C header 44156d84838 Kernel: Add chainstate loading to kernel C header 2cee46cdcc1 kernel: Add chainstate manager object to C header 7102c7ae45e kernel: Add notifications context option to C header ed628a2a3c4 kerenl: Add chain params context option to C header 27643297ff7 kernel: Add kernel library context object 2ba22cf3f90 kernel: Add logging to kernel library C header 873874c03e9 kernel: Introduce initial kernel C header API d94adc7270b Merge bitcoin/bitcoin#29702: fees: Remove CLIENT_VERSION serialization 7290bc61c00 Merge bitcoin/bitcoin#31078: build: Fix kernel static lib component install 68f29b24907 Merge bitcoin/bitcoin#31141: doc: Make list of targets in depends README consistent e9b95665eea Merge bitcoin/bitcoin#31046: init: Some small chainstate load improvements b8c821cc1ea Merge bitcoin/bitcoin#30724: test: add test for specifying custom pidfile via `-pid` a0c9595810c doc: Make list of targets in depends README consistent fa1c5cc9df1 fees: Log non-fatal errors as [warning], instead of info-level ffe4261cb06 Merge bitcoin/bitcoin#30935: ci: Approximate MAKEJOBS in image build phase 28ce159bc32 Merge bitcoin/bitcoin#30183: rpc: net: follow-ups for #30062 684873931b3 Merge bitcoin/bitcoin#26334: Add Signet and testnet4 launch shortcuts for Windows 9b0e2598089 Merge bitcoin/bitcoin#31121: guix: Enable CET for `glibc` package d9f8dc64534 Merge bitcoin/bitcoin#31097: validation: Improve input script check error reporting a16917fb598 rpc, net: improve `mapped_as` doc for getrawaddrman/getpeerinfo 563c4d29268 Merge bitcoin/bitcoin#31105: Update libmultiprocess library 0e9f20625a1 Merge bitcoin/bitcoin#31063: lint: commit-script-check.sh: echo to stderr e8f72aefd20 Merge bitcoin/bitcoin#29877: tracing: explicitly cast block_connected duration to nanoseconds 86e2a6b749c [test] A non-standard transaction which is also consensus-invalid should return the consensus error 4d3da08d1b9 guix: Enable CET for `glibc` package a38603456e9 Merge bitcoin/bitcoin#31100: doc: remove dependency install instructions from win docs 90b405516f7 Update libmultiprocess library 479715e9db0 Merge bitcoin/bitcoin#30996: doc: update signet documentation related to build directories 99e041f86fd Merge bitcoin/bitcoin#31099: doc: drop macOS LLVM install instructions 21e2f06a1cc Merge bitcoin/bitcoin#31067: test: Print CompletedProcess object on error 184f12c1542 doc: remove dependency install instructions from win docs dea9fb9a8b8 Merge bitcoin/bitcoin#30093: optimization: reserve memory allocation for transaction inputs/outputs 79aa8280b2e doc: drop LLVM install instructions 2123c94448e Merge bitcoin/bitcoin#30527: Bump python minimum supported version to 3.10 538ccaed004 Merge bitcoin/bitcoin#31048: build: Bump minimum supported macOS to 13.0 f859ff8a4e9 [validation] Improve script check error reporting ddddbac9c10 fees: Pin required version to 149900 fa5126adcb1 fees: Pin "version that wrote" to 0 0ca1d1bf69c Merge bitcoin/bitcoin#31092: doc: fuzz: remove Honggfuzz NetDriver instructions d823ba6e20b doc: fuzz: remove Honggfuzz NetDriver instructions 15563d3388e Merge bitcoin/bitcoin#30859: doc: cmake: prepend "build" to functional/test_runner.py 2ac5ba24bf0 Merge bitcoin/bitcoin#31083: doc: add doxygen for m_args in tests a0e089a71dc build: Bump minimum supported macOS to 13.0 1fe1b3ba8e9 doc: doxygen comment for m_args usage in tests 82e16e69832 cmake: Refactor install kernel dependencies 42e62779873 build: Add static libraries to Kernel install component e64b2f1a16e doc: cmake: prepend and explain "build/" where needed 48cf3da6360 Merge bitcoin/bitcoin#30970: build: Add missing USDT header dependency to kernel d8b835cf18c Merge bitcoin/bitcoin#31070: contrib: fix typos in check-deps.sh da8824ba301 Fix typos in check-deps.sh fa43c4f93ca test: Print CompletedProcess object on error 489e5aa3a29 Merge bitcoin/bitcoin#30857: cluster mempool: extend DepGraph functionality 9f45062b9b0 Merge bitcoin/bitcoin#30937: build: scripted-diff: drop config/ subdir for bitcoin-config.h 882f736d0a6 doc: lint: correct outdated comment (s/Makefile.am/CMakeLists.txt/) 1786be7b4a5 scripted-diff: drop config/ subdir for bitcoin-config.h, rename to bitcoin-build-config.h 0c2c3bb3f5c Merge bitcoin/bitcoin#30955: Mining interface: getCoinbaseMerklePath() and submitSolution() 9909a34d794 Merge bitcoin/bitcoin#30992: doc: update IBD requirements in doc/README.md fac6cfe5ac0 lint: commit-script-check.sh: echo to stderr 5fb94550638 Merge bitcoin/bitcoin#31058: refactor: include the proper header rather than forward-declaring RemovalReasonToString e569eb8d917 Merge bitcoin/bitcoin#30885: scripted-diff: Modernize nLocalServices naming 31cc5006c3d init: Return fatal failure on snapshot validation failure 8f1246e8338 init: Improve chainstate init db error messages 5837e3463fe Merge bitcoin/bitcoin#30967: refactor: Replace g_genesis_wait_cv with m_tip_block_cv ca2e4ba352c refactor: include the proper header rather than forward-declaring RemovalReasonToString a9f6a57b691 Merge bitcoin/bitcoin#30920: test: Remove 0.16.3 test from wallet_backwards_compatibility.py fa71bedf860 ci: Approximate MAKEJOBS in image build phase 03696bb1bd5 Merge bitcoin/bitcoin#31045: ci: Add missing -DWERROR=ON to test-each-commit 56093565bbe Merge bitcoin/bitcoin#31018: test: Treat exclude list warning as failure in CI bb47b5a6576 Merge bitcoin/bitcoin#31038: test: Fix copy-paste in wallet/test/db_tests ostream operator 3fecf36c7b3 Merge bitcoin/bitcoin#31056: ci: Double ctest timeout 3c4a9419dbe Merge bitcoin/bitcoin#31013: depends: For mingw cross compile use -gcc-posix to prevent library conflict 5d5cc021ce3 Merge bitcoin/bitcoin#31051: test: remove unused code from `script_tests` caf44e500eb Merge bitcoin/bitcoin#31008: depends: Print ready-to-use `--toolchain` option for CMake invocation fa5ebc99207 ci: Double ctest timeout 0b3ec8c59b2 clusterlin: remove Cluster type 1c24c625105 clusterlin: merge two DepGraph fuzz tests into simulation test 0606e66fdbb clusterlin: add DepGraph::RemoveTransactions and support for holes in DepGraph 75b5d42419e clusterlin: make DepGraph::AddDependency support multiple dependencies at once abf50649d13 clusterlin: simplify DepGraphFormatter::Ser eaab55ffc81 clusterlin: rework DepGraphFormatter::Unser 5901cf7100a clusterlin: abstract out DepGraph::GetReduced{Parents,Children} e0287bc4b2d test: remove unused code from script_tests 62e45167221 Merge bitcoin/bitcoin#31026: ci: set a ctest test timeout of 1200 (20 minutes) cd093049dda init: Remove incorrect comment about shutdown condition 635e9f85d76 init: Remove misleading log line when user chooses not to retry fa1cffacae6 ci: Install missing nproc in macos task faf7a2bccc7 ci: Add missing -DWERROR=ON to test-each-commit 56aad83307e ci: set a ctest timeout of 1200 (20 minutes) 1b707146717 Merge bitcoin-core/gui#840: qt6: Handle different signatures of `QANEF::nativeEventFilter` 720ce880a35 init: Improve comment describing chainstate load retry behaviour baea842ff18 init: Remove unneeded argument for mempool_opts checks ec58dfe8f74 Merge bitcoin/bitcoin#31010: cmake: Avoid hardcoding Qt's major version in Find module / variable names 5fe6878b5f7 Merge bitcoin-core/gui#836: Fix display issues for IPv6 proxy setup in Options Dialog (UI only, no functionality impact) f50557f5d36 test: Fix copy-paste in db_tests ostream operator 5ea335a97f8 Merge bitcoin/bitcoin#30793: rpc: add getorphantxs 76e2e8aabd8 Merge bitcoin/bitcoin#31035: doc: Archive 28.0 release notes f019fcec412 doc: Archive 28.0 release notes 80761afced1 qt6: Handle different signatures of `QANEF::nativeEventFilter` 51c698161b5 Merge bitcoin-core/gui#837: qt6: Fix linking when configured with `-DENABLE_WALLET=OFF` 4be785b3e33 Merge bitcoin-core/gui#839: qt6, test: Handle deprecated code f117f3f7473 Merge bitcoin-core/gui#838: qt6: Handle deprecated `QLocale::nativeCountryName` 5625840c11d qt6, test: Handle deprecated `QVERIFY_EXCEPTION_THROWN` 772928a13c2 Merge bitcoin/bitcoin#30982: docs: Add instructions on how to self-sign bitcoin-core binaries for macOS 27709f51ee0 docs: Add instructions on how to self-sign bitcoin-core binaries for macOS cfb59da4b3b Merge bitcoin/bitcoin#30980: fuzz: fix bug in p2p_headers_presync harness dda2613239b Merge bitcoin/bitcoin#30929: log: Enforce trailing newline e0ae9c14c4e Merge bitcoin/bitcoin#31011: refactor: move util/pcp and util/netif to common/ 98c1536852d test: add getorphantxs tests 93f48fceb7d test: add tx_in_orphanage() 34a9c10e8cd rpc: add getorphantxs f511ff3654d refactor: move verbosity parsing to rpc/util 36a6d4b0078 doc: update IBD requirements in doc/README.md fa6d14eacb2 test: Treat exclude list warning as failure in CI 6a370435526 Merge bitcoin/bitcoin#31007: doc: add testnet4 section header for config file 70910eb2ecb Merge bitcoin/bitcoin#31016: test: add missing sync to feature_fee_estimation.py 532491faf1a net: add GetOrphanTransactions() to PeerManager 91b65adff2a refactor: add OrphanTxBase for external use a1576edab35 test: add missing sync to feature_fee_estimation.py ae56b3230b2 depends: For mingw cross compile use -gcc-posix to prevent library conflict fd38711217c ci: make CI job fail when check-deps.sh script fails d51edecddcb common: move pcp.cpp and netif.cpp files from util to common library since they depend on netaddress.cpp 61cdb1c9d83 doc: add testnet4 section header for config file deacf3c7cd6 cmake: Avoid hardcoding Qt's major version in Find module 605926da0ab depends: Print ready-to-use `--toolchain` option for CMake invocation fa2b7d8d6b3 Remove redundant unterminated-logprintf tidy check bbbb2e43ee9 log: Enforce trailing newline, Remove redundant m_started_new_line fa22e5c430a refactor: Remove dead code that assumed tip == nullptr fa2e4439652 refactor: Replace g_genesis_wait_cv with m_tip_block_cv fa7f52af1a4 refactor: Use wait_for predicate to check for interrupt 5ca28ef28bc refactor: Split up NodeContext shutdown_signal and shutdown_request fad8e7fba7b bugfix: Mark m_tip_block_cv as guarded by m_tip_block_mutex fa18586c29d refactor: Add missing GUARDED_BY(m_tip_block_mutex) fa4c0750331 doc: Clarify waitTipChanged docs fc642c33ef2 Merge bitcoin/bitcoin#30718: test: switch MiniWallet padding unit from weight to vsize d7f956a309e Merge bitcoin/bitcoin#30968: init: Remove retry for loop c33eb2360e2 Merge bitcoin/bitcoin#30043: net: Replace libnatpmp with built-in PCP+NATPMP implementation f3c74c4a7e1 Merge bitcoin/bitcoin#30989: guix: Drop no longer needed `PATH` modification 5c7cacf649a ci: Remove natpmp build option and libnatpmp dependency 7e7ec984da5 doc: Remove mention of natpmp build options 061c3e32a26 depends: Drop natpmp and associated option from depends 20a18bf6aa3 build: Drop libnatpmp from build system 7b04709862f qt: Changes for built-in PCP+NAT-PMP 52f8ef66c61 net: Replace libnatpmp with built-in NATPMP+PCP implementation in mapport 97c97177cdb net: Add PCP and NATPMP implementation cb750b4b405 qt6, test: Use `qWarning()` instead of `QWARN()` macro 9123a286e97 qt6: Handle deprecated `QLocale::nativeCountryName` 940edd6ac24 test: refactor: introduce and use `TRUC_CHILD_MAX_VSIZE` constant c16ae717689 test: switch MiniWallet padding unit from weight to vsize a647d4400d5 doc: update signet documentation related to build directories f1daa80521e guix: Drop no longer needed `PATH` modification d812cf11896 Merge bitcoin/bitcoin#30879: test: re-bucket long-running tests 18d4c43cab4 Merge bitcoin/bitcoin#30921: test: generalize HasReason and use it in FailFmtWithError d7fcc91416a Merge bitcoin/bitcoin#30974: ci: Inline PACKAGE_MANAGER_INSTALL 29d00a1cee1 Merge bitcoin/bitcoin#30940: depends: Fix build with `MULTIPROCESS=1` in Guix environment 89a8e9b732f Merge bitcoin/bitcoin#30979: contrib: Update asmap link in seeds readme fafd1a0f648 ci: Inline PACKAGE_MANAGER_INSTALL 36ad9516dbd Merge bitcoin/bitcoin#30981: ci: add timestamps to cirrus jobs fa7c2838a5f Merge bitcoin/bitcoin#30948: test: Add missing sync_mempools() to fill_mempool() f951f1fab25 ci: add timestamps to cirrus jobs f158993fd55 contrib: Update asmap link in seeds readme d5af7d28f47 Merge bitcoin/bitcoin#30976: depends, doc: Drop package-specific note about CMake a7498cc7e26 Fix bug in p2p_headers_presync harness 4cf84b344de depends, doc: No need to specify general requirement e13da501db9 Merge bitcoin/bitcoin#30973: doc: fix `loadtxoutset` example 513b7136c79 Merge bitcoin/bitcoin#30961: ci: add `LLVM_SYMBOLIZER_PATH` to Valgrind fuzz job 286725168ae doc: fix loadtxoutset example 525e9dcba0b Add submitSolution to BlockTemplate interface 47b4875ef05 Add getCoinbaseMerklePath() to Mining interface 63d6ad7c89c Move BlockMerkleBranch back to merkle.{h,cpp} 65f6e7078b1 Merge bitcoin/bitcoin#30510: multiprocess: Add IPC wrapper for Mining interface da612cea032 Merge bitcoin/bitcoin#30962: validation: Disable CheckForkWarningConditions for background chainstate e9d60af9889 refactor: Replace init retry for loop with if statement c1d8870ea41 refactor: Move most of init retry for loop to a function ccd10fdb97f build: Add missing USDT header dependency to kernel 781c01f5806 init: Check mempool arguments in AppInitParameterInteractions 39219fe145e Merge bitcoin/bitcoin#30946: doc: correct the zmq automatic build info 06e7e836329 doc: correct the zmq automatic build info a9773b6215e Merge bitcoin/bitcoin#30963: doc: Adjust links in OSS-Fuzz section fa6c1946d23 doc: Adjust links in OSS-Fuzz section c0a0c72b4d6 validation: Disable CheckForkWarningConditions for background chainstate c1832584bfd ci: add LLVM_SYMBOLIZER_PATH to Valgrind fuzz job 393f323bd60 Merge bitcoin/bitcoin#30952: test: Use shell builtins in run_command test case faf801515f8 test: Add missing sync_mempools() to fill_mempool() fa48be6f023 test: Refactor fill_mempool to extract send_batch helper 1a332817665 doc: multiprocess documentation improvements 90a5786bba4 Merge bitcoin/bitcoin#30678: wallet: Write best block to disk before backup d043950ba24 multiprocess: Add serialization code for BlockValidationState 33c2eee285e multiprocess: Add IPC wrapper for Mining interface 06882f84017 multiprocess: Add serialization code for vector<char> 095286f790a multiprocess: Add serialization code for CTransaction 69dfeb18761 multiprocess: update common-types.h to use C++20 concepts 206c6e78eec build: Make bitcoin_ipc_test depend on bitcoin_ipc 070e6a32d5f depends: Update libmultiprocess library for cmake headers target dabc74e86c3 Merge bitcoin/bitcoin#30409: Introduce waitTipChanged() mining interface, replace RPCNotifyBlockChange, drop CRPCSignals & g_best_block 7bd3ee62f6d test: Use shell builtins in run_command test case f5a2000579b test: re-bucket long-running tests 06b4c339e89 depends: Fix reproducibility when building with `MULTIPROCESS=1` 04e4d52420a test: add test for specifying custom pidfile via `-pid` b832ffe0446 refactor: introduce default pid file name constant in tests d8e3afc3352 depends: Fix build with `MULTIPROCESS=1` in Guix environment f20fe33e94c test: Add basic balance coverage to wallet_assumeutxo.py d72df63d169 net: Use GetLocalAddresses in Discover e02030432b7 net: Add netif utility 754e4254388 crypto: Add missing WriteBE16 function 33adc7521cc Merge bitcoin/bitcoin#30765: refactor: Allow `CScript`'s `operator<<` to accept spans, not just vectors 0894748316c Merge bitcoin/bitcoin#30918: fuzz: Add check in `p2p_headers_presync` that chain work never exceeds minimum work f57a6754ed6 Merge bitcoin/bitcoin#30826: fuzz: reduce number of iterations in `crypto_aeadchacha20poly1305` target 48c20dbd86c Merge bitcoin/bitcoin#30794: interpreter: use int32_t instead of int type for risczero compile 4148e60909e Merge bitcoin/bitcoin#30679: fix: handle invalid `-rpcbind` port earlier a8a2628b7a9 Merge bitcoin/bitcoin#30828: interfaces: #30697 follow ups 0d81b3ddedc Merge bitcoin/bitcoin#30568: addrman: change internal id counting to int64_t c985a34b9c3 Merge bitcoin/bitcoin#26990: cli: Improve error message on multiwallet cli-side commands 037b101e808 test: Add coverage for best block locator write in wallet_backup 31c0df03890 wallet: migration, write best locator before unloading wallet 7e3dbe4180c wallet: Write best block to disk before backup 79f20fa1b1e Merge bitcoin/bitcoin#30561: refactor: move `SignSignature` helpers to test utils 284bd17309a add check that chainwork doesn't exceed minimum work 9aa5d1c3fcd add clarification in comment 197aa249551 Merge bitcoin/bitcoin#30856: build: drop obj/ subdirectory for generated build.h 7025942687f build: drop superfluous `HAVE_BUILD_INFO` define 0dd662510c5 build: drop obj/ subdir for generated build.h, rename to bitcoin-build-info.h 84cd6478c42 Merge bitcoin/bitcoin#30927: Follow-up after AutoFile position caching: remove unused code caac06f784c streams: reorder/document functions 67a3d590768 streams: remove unused code 2db926f49c8 Merge bitcoin/bitcoin#30889: log: Use ConstevalFormatString fee4cba4847 gui: Fix proxy details display in Options Dialog 9ba56884f62 Merge bitcoin/bitcoin#30869: ci: Print inner env, Make ccache config more flexible 6c3c619b35c test: generalize HasReason and use it in FailFmtWithError ab0b5706b25 Merge bitcoin/bitcoin#30875: doc: fixed inconsistencies in documentation between autotools to cmake change fd08fded63a Merge bitcoin/bitcoin#30639: ci: Use clang-19 in msan tasks 5be34bacf6d qt: Fix linking when configured with `-DENABLE_WALLET=OFF` a9964c04447 doc: Updating docs from autotools to cmake fae44c83da9 test: Remove 0.16.3 test from wallet_backwards_compatibility.py 69409bc6e55 Merge bitcoin/bitcoin#30908: doc: remove Eclipser fuzzing documentation 6b97882ab53 Merge bitcoin/bitcoin#30915: ci: Use `ninja` to build in macOS native CI job e6994efe08b fix: increase rpcbind check robustness d38e3aed89e fix: handle invalid rpcbind port earlier 83b67f2e6d5 refactor: move host/port checking 73c243965ab test: add tests for invalid rpcbind ports 54227e681a4 rpc, cli: improve error message on multiwallet mode 735436df8ce Remove outdated Eclipser fuzzing documentation ccccb67851b ci: Use clang-19 in msan tasks facbcd4cef8 log: Use ConstevalFormatString d01b85bfecb ci: Use `ninja` to build in macOS native CI job 6fc46927971 Merge bitcoin/bitcoin#29624: doc: update NeedsRedownload() and nStatus comment 2a0949f0977 Merge bitcoin/bitcoin#30888: build: optimize .h generation in GenerateHeaderFrom{Raw,Json}.cmake bdbc90f29ac Merge bitcoin/bitcoin#30902: Remove Autotools packages from CI (and depends doc) a95e742b692 Merge bitcoin/bitcoin#30913: ci: Use macos-14 GHA image (x86_64-apple-darwin22.6.0 -> arm64-apple-darwin23.6.0) 225718eda89 Merge bitcoin/bitcoin#30438: guix: (explicitly) build Linux GCC with `--enable-cet` fab932b4211 ci: Remove incorrectly hardcoded HOST in mac_native task af9f9878934 doc: update NeedsRedownload() comment fa8f35d7865 ci: Use macos-14 GHA image 7942951e3fc Remove unused g_best_block e3a560ca68d rpc: use waitTipChanged for longpoll 460687a09c2 Remove unused CRPCSignals dca923150e3 Replace RPCNotifyBlockChange with waitTipChanged() 2a40ee11219 rpc: check for negative timeout arg in waitfor* de7c855b3af rpc: recommend -rpcclienttimeout=0 for waitfor* 77ec072925a rpc: fix waitfornewblock description 285fe9fb51c rpc: add test for waitforblock and waitfornewblock b94b27cf05c Add waitTipChanged to Mining interface 7eccdaf1608 node: Track last block that received a blockTip notification ebb8215f236 Rename getTipHash() to getTip() and return BlockRef 89a8f74bbbb refactor: rename BlockKey to BlockRef 9f1aa88d4d9 Merge bitcoin/bitcoin#30884: streams: cache file position within AutoFile 06329eb1348 Merge bitcoin/bitcoin#29436: net: call `Select` with reachable networks in `ThreadOpenConnections` e983ed41d9f Merge bitcoin/bitcoin#30410: rpc, rest: Improve block rpc error handling, check header before attempting to read block data. fce9e065c16 Merge bitcoin/bitcoin#28358: Drop -dbcache limit 8d000b85dd4 Merge bitcoin/bitcoin#30868: refactor: add clang-tidy `modernize-use-starts-ends-with` check 3f66642820b Merge bitcoin/bitcoin#30440: Have createNewBlock() return a BlockTemplate interface 2bf721e76a5 Merge bitcoin/bitcoin#30661: fuzz: Test headers pre-sync through p2p c38e9993de7 Merge bitcoin/bitcoin#30286: cluster mempool: optimized candidate search fa99e4521b6 ci: Allow CCACHE_DIR bind mount 37679b856ce Merge bitcoin/bitcoin#30899: qt: Translations update fc7b507e9a5 tidy: add clang-tidy `modernize-use-starts-ends-with` check 7a8a6a06676 doc: Fix comment in `contrib/devtools/check-deps.sh` script 712d105e093 depends, doc: Do not install Autotools packages b786449e663 ci: Do not install Autotools packages a240e150e83 streams: remove AutoFile::Get() entirely ae052957614 qt: Translations update 6a1aa510e31 rpc: check block index before reading block / undo data 6cbf2e5f819 rpc: Improve gettxoutproof error when only header is available. 69fc867ea19 test: add coverage to getblock and getblockstats 5290cbd5850 rpc: Improve getblock / getblockstats error when only header is available. e5b537bbdfb rest: improve error when only header of a block is available. e624a9bef16 streams: cache file position within AutoFile 89bf11b8072 guix: build Linux GCC with --enable-cet a93c171faa7 Drop unneeded nullptr check from CreateNewBlock() dd87b6dff35 Have createNewBlock return BlockTemplate interface fae9b60c4ff test: Use LogPrintStr to test m_log_sourcelocations 33381ea530a scripted-diff: Modernize nLocalServices to m_local_services 2a581144f28 build: Minimize I/O operations in GenerateHeaderFromJson.cmake aa003d1568b build: Minimize I/O operations in GenerateHeaderFromRaw.cmake 9ad2fe7e69e clusterlin: only start/use search when enough iterations left bd044356edb clusterlin: improve heuristic to decide split transaction (optimization) 71f26293988 clusterlin: include topological pot subsets automatically (optimization) e20fda77a2d clusterlin: reduce computation of unnecessary pot sets (optimization) 6060a948caf clusterlin bench: add example hard cluster benchmarks 2965fbf203f clusterlin: track upper bound potential set for work items (optimization) 9e43e4ce109 clusterlin: use feerate-sorted depgraph in SearchCandidateFinder b80e6dfe780 clusterlin: add reordering support for DepGraph 85a285a3061 clusterlin: separate initial search entries per component (optimization) e4faea9ca79 clusterlin bench: have low/high iter benchmarks instead of per-iter fa39b1ca638 doc: move-only logging warning fa252da0b9c ci: Remove hardcoded CCACHE_DIR in cirrus fa146904e19 ci: Bump default CCACHE_MAXSIZE to 500M aaaa7cf8bad cirrus: Drop CCACHE_NOHASHDIR fa7ca182a9b ci: Print inner env 84663291275 chain: simplify `deleteRwSettings` code and improve it's doc f8d91f49c70 chain: dont check for null settings value in `overwriteRwSetting` df601993f2d chain: ensure `updateRwSetting` doesn't update to a null settings 5e190cd11f6 Replace CScript _hex_v_u8 appends with _hex cac846c2fbf Allow CScript's operator<< to accept spans, not just vectors c78d8ff4cb8 prevector: avoid GCC bogus warnings in insert method e4e3b44e9cc net: call `Select` with reachable networks in `ThreadOpenConnections` 829becd990b addrman: change `Select` to support multiple networks f698636ec86 net: add `All()` in `ReachableNets` a97f43d63a6 fuzz: Add harness for p2p headers sync c8e2eeeffb4 chain: uniformly use `SettingsAction` enum in settings methods f482d0e366a fuzz: reduce number of iterations in `crypto_aeadchacha20poly1305` target 1e9e735670f chain: move new settings safely in `overwriteRwSetting` 1c409004c80 test: remove wallet context from `write_wallet_settings_concurrently` 58499b00d0a refactor: move `SignSignature` helpers to test utils cd0edf26c07 tracing: cast block_connected duration to nanoseconds cfd03de965a Add Testnet4 launch shortcut for Windows 77b2923f871 Add Signet launch shortcut for Windows bc52cda1f3c fix use int32_t instead of int type for risczero compile with (-march=rv32i, -mabi=ilp32) a0eaa4749fe Add FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION in PoW check a3f6f5acd89 build: Automatically define FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION for fuzz builds 0c02d4b2bdb net_processing: Make MAX_HEADERS_RESULTS a PeerManager option 51f7668d31e addrman: change nid_type from int to int64_t 051ba3290e3 addrman, refactor: introduce user-defined type for internal nId bdad0243be8 rpc, net: getrawaddrman "mapped_as" follow-ups fa1b139d17d Bump python minimum supported version to 3.10 ec585f11c38 Reserve space for transaction inputs in CreateTransactionInternal c76aaaf9003 Reserve space for transaction outputs in CreateTransactionInternal bb3b980dfd9 validation: drop maximum -dbcache REVERT: e70e527ef21 kernel: Add pure kernel bitcoin-chainstate REVERT: ed9a8a54d3c kernel: Add block index utility functions to C header REVERT: 6338dd45b55 kernel: Add function to read block undo data from disk to C header REVERT: a2ac0c1e7c9 kernel: Add functions to read block from disk to C header REVERT: 170060c3372 kernel: Add function for copying block data to C header REVERT: 0f8c00bba07 kernel: Add functions for the block validation state to C header REVERT: 41dba7d2603 kernel: Add validation interface to C header REVERT: 877cf01f22c kernel: Add interrupt function to C header REVERT: f77c2b90422 kernel: Add import blocks function to C header REVERT: 254e17dbeab kernel: Add chainstate load options for in-memory dbs in C header REVERT: 8baa06d318f kernel: Add options for reindexing in C header REVERT: 0243ed8a200 kernel: Add block validation to C header REVERT: 36fbab87e9e Kernel: Add chainstate loading to kernel C header REVERT: b249e93f295 kernel: Add chainstate manager object to C header REVERT: 2546745a393 kernel: Add notifications context option to C header REVERT: 2578746e87f kerenl: Add chain params context option to C header REVERT: 21107de0ca7 kernel: Add kernel library context object REVERT: 83cc65c4911 kernel: Add logging to kernel library C header REVERT: 2f47169f91e kernel: Introduce initial kernel C header API git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 1047757ea3b4b78b51d7338ea44e2123851143fe

davidgumberg mentioned this pull request Sep 12, 2024

28.0rc1 synchronizes much slower on Windows #30833

Closed

sipa force-pushed the 202409_reduce_ftell_xor branch from 4a37953 to 8a7d8f7 Compare September 12, 2024 19:22

DrahtBot added the CI failed label Sep 12, 2024

sipa force-pushed the 202409_reduce_ftell_xor branch from 8a7d8f7 to 4cfff4e Compare September 12, 2024 21:41

davidgumberg reviewed Sep 13, 2024

View reviewed changes

DrahtBot mentioned this pull request Sep 13, 2024

scripted-diff: Use LogInfo over LogPrintf [WIP, NOMERGE, DRAFT] #29641

Draft

davidgumberg reviewed Sep 13, 2024

View reviewed changes

DrahtBot mentioned this pull request Sep 13, 2024

util: explicitly close all AutoFiles that have been written #29307

Merged

davidgumberg reviewed Sep 13, 2024

View reviewed changes

src/streams.cpp Outdated Show resolved Hide resolved

DrahtBot mentioned this pull request Sep 13, 2024

Embed default ASMap as binary dump header file #28792

Open

davidgumberg reviewed Sep 13, 2024

View reviewed changes

Oaksta85 approved these changes Sep 13, 2024

View reviewed changes

sipa force-pushed the 202409_reduce_ftell_xor branch from 4cfff4e to 82183ee Compare September 13, 2024 11:28

streams: cache file position within AutoFile

e624a9b

sipa force-pushed the 202409_reduce_ftell_xor branch from 82183ee to e624a9b Compare September 13, 2024 11:35

fanquake linked an issue Sep 13, 2024 that may be closed by this pull request

28.0rc1 synchronizes much slower on Windows #30833

Closed

fanquake added this to the 28.0 milestone Sep 13, 2024

fanquake added the Needs backport (28.x) label Sep 13, 2024

DrahtBot removed the CI failed label Sep 13, 2024

DrahtBot requested a review from davidgumberg September 13, 2024 16:13

streams: remove AutoFile::Get() entirely

a240e15

Co-Authored-By: David Gumberg <davidzgumberg@gmail.com>

theStack approved these changes Sep 15, 2024

View reviewed changes

DrahtBot requested a review from achow101 September 15, 2024 23:16

davidgumberg reviewed Sep 16, 2024

View reviewed changes

DrahtBot requested a review from davidgumberg September 16, 2024 21:45

achow101 merged commit 9f1aa88 into bitcoin:master Sep 17, 2024
16 checks passed

achow101 pushed a commit to achow101/bitcoin that referenced this pull request Sep 17, 2024

streams: cache file position within AutoFile

8229e98

Github-Pull: bitcoin#30884 Rebased-From: e624a9b

achow101 pushed a commit to achow101/bitcoin that referenced this pull request Sep 17, 2024

streams: remove AutoFile::Get() entirely

ff95cb3

Co-Authored-By: David Gumberg <davidzgumberg@gmail.com> Github-Pull: bitcoin#30884 Rebased-From: a240e15

achow101 mentioned this pull request Sep 17, 2024

[28.x] Further backports and rc2 #30827

Merged

fanquake removed the Needs backport (28.x) label Sep 17, 2024

maflcko reviewed Sep 19, 2024

View reviewed changes

sipa mentioned this pull request Sep 19, 2024

Follow-up after AutoFile position caching: remove unused code #30927

Merged

l0rinc mentioned this pull request Mar 12, 2025

[IBD] Tracking PR for speeding up Initial Block Download #32043

Draft

11 tasks

		} else if (std::ferror(afile.Get())) {
		LogPrintf("[snapshot] warning: i/o error reading %s\n", read_from_str);

streams: cache file position within AutoFile #30884

streams: cache file position within AutoFile #30884

Uh oh!

Conversation

sipa commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage

Reviews

Conflicts

Uh oh!

davidgumberg commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipa commented Sep 12, 2024

Uh oh!

vostrnad commented Sep 12, 2024

Uh oh!

davidgumberg Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sipa Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidgumberg Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laanwj Sep 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidgumberg commented Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Why the regression happened

Footnotes

Uh oh!

sipa commented Sep 13, 2024

Uh oh!

laanwj commented Sep 13, 2024

Uh oh!

achow101 commented Sep 13, 2024

Uh oh!

sipa commented Sep 14, 2024

Uh oh!

vostrnad commented Sep 15, 2024

Uh oh!

theStack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

achow101 commented Sep 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidgumberg commented Sep 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

achow101 commented Sep 17, 2024

Uh oh!

sipa commented Sep 12, 2024 •

edited

Loading

DrahtBot commented Sep 12, 2024 •

edited

Loading

davidgumberg commented Sep 12, 2024 •

edited

Loading

davidgumberg Sep 13, 2024 •

edited

Loading

sipa Sep 13, 2024 •

edited

Loading

davidgumberg Sep 13, 2024 •

edited

Loading

laanwj Sep 17, 2024 •

edited

Loading

davidgumberg commented Sep 13, 2024 •

edited

Loading

davidgumberg commented Sep 16, 2024 •

edited

Loading

maflcko Sep 19, 2024 •

edited

Loading