Cocnept ACK, this is great!

Note that all dependencies do compile, except for Qt (build issue with FreeType I have not investigated), I've just excluded all optional dependencies from the depends build above.

Qt is going to be a big challenge for full-static linking (due to system dependencies for X, FreeType etc). It's ok to skip that for now, IMO, would be nice to be able to ship a statically linked bitcoind and utilities at least.

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #16545 (refactor: Implement missing error checking for ArgsManager flags by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Looking at the depends build log, I think the custom toolchain isn't correctly fed into the boost build. It's using my system compiler:
(snip—this was wrong)

It's using my system compiler:

That excerpt is for the native b2 build, where using the system compiler is expected.

I might have not properly cleaned my tree. Will retry a full build.

Edit: whoops, that did it.
Testing note make sure you start from a clean tree. If you get a linking error about boost::system::generic_category(), this is caused by pollution from previous builds.

Great, the functional tests pass and so does test_bitcoin
make check fails on the secp256k1 tests, but not because of any error, simply because they're not built.

FAIL: tests
===========

/home/user/src/bitcoin/src/secp256k1/build-aux/test-driver: line 107: ./tests: No such file or directory
FAIL tests (exit status: 127)

FAIL: exhaustive_tests
======================

/home/user/src/bitcoin/src/secp256k1/build-aux/test-driver: line 107: ./exhaustive_tests: No such file or directory
FAIL exhaustive_tests (exit status: 127)

They do exist, but it's unable to run them somehow;

~/src/bitcoin$ ls -al build/src/secp256k1/exhaustive_tests build/src/secp256k1/tests
-rwxr-xr-x 1 user user 128480 Oct  6 16:15 build/src/secp256k1/exhaustive_tests
-rwxr-xr-x 1 user user 466624 Oct  6 16:15 build/src/secp256k1/tests
~/src/bitcoin$ build/src/secp256k1/tests
bash: build/src/secp256k1/tests: No such file or directory
~/src/bitcoin$ file build/src/secp256k1/tests
build/src/secp256k1/tests: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, not stripped

I think this is the problem: "interpreter /lib/ld-musl-x86_64.so.1"

The build works as-is on RISC-V! Except I had to add --without-asm to prevent secp256k1 from including x86 assembly (failing while linking bitcoind etc). I think the problem is that the instruction set detection in secp256k1's configure compiles a C file with __asm__ in it but doesn't link it: the IR files likely contain the assembly code as text, and only in the link phase assembling is actually attempted. So it doesn't detect that x86 assembly isn't valid for the platform.

Edit: also the tests segfault on RISC-V, no idea why yet. Ok, built with debug information, this is the traceback:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x0000000000519e74 in std::__glibcxx_rwlock_rdlock (__rwlock=0xaebdd8 <_ZN12_GLOBAL__N_1L14signatureCacheE.lto_priv.0+304>) at /usr/include/c++/10/shared_mutex:73
#2  std::__shared_mutex_pthread::lock_shared (this=0xaebdd8 <_ZN12_GLOBAL__N_1L14signatureCacheE.lto_priv.0+304>) at /usr/include/c++/10/shared_mutex:224
#3  std::shared_mutex::lock_shared (this=<optimized out>, this=<optimized out>) at /usr/include/c++/10/shared_mutex:421
#4  std::shared_lock<std::shared_mutex>::shared_lock (__m=..., this=<optimized out>, this=<optimized out>, __m=...) at /usr/include/c++/10/shared_mutex:722
#5  (anonymous namespace)::CSignatureCache::Get (this=<optimized out>, erase=true, entry=...) at script/sigcache.cpp:69
#6  CachingTransactionSignatureChecker::VerifyECDSASignature (this=0x3ff5049998, vchSig=..., pubkey=..., sighash=...) at script/sigcache.cpp:109
#7  0x00000000005c6082 in GenericTransactionSignatureChecker<CTransaction>::CheckECDSASignature (this=0x3ff5049998, vchSigIn=..., vchPubKey=..., scriptCode=..., sigversion=<optimized out>) at script/interpreter.cpp:1688
#8  0x00000000006c49e2 in EvalChecksigPreTapscript (fSuccess=@0x3ff5049179: true, serror=0x3ff0000bcc, sigversion=<optimized out>, checker=..., flags=134673, pend=..., pbegincodehash=..., vchPubKey=..., vchSig=...) at script/interpreter.cpp:363
#9  EvalChecksig(std::vector<unsigned char, std::allocator<unsigned char> > const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, prevector<28u, unsigned char, unsigned int, int>::const_iterator, prevector<28u, unsigned char, unsigned int, int>::const_iterator, ScriptExecutionData&, unsigned int, BaseSignatureChecker const&, SigVersion, ScriptError_t*, bool&) [clone .constprop.0] (sig=..., pubkey=..., pbegincodehash=..., pend=..., execdata=..., flags=<optimized out>, checker=..., sigversion=<optimized out>,
    serror=0x3ff0000bcc, success=@0x3ff5049179: true) at script/interpreter.cpp:421
#10 0x00000000005cf5de in EvalScript (stack=..., script=..., flags=<optimized out>, checker=..., sigversion=<optimized out>, execdata=..., serror=0x3ff0000bcc) at script/interpreter.cpp:1094

I'm not sure what this is. But I strongly suspect it's not a bug in our code. It could be some kind of C++ ABI conflict (does it matter what C++ libarary version is on the system?), but also simply a RISC-V specific compilation / library /thread handling bug. We'll know more once we try on for other platforms.

Concept ACK, always great to see people doing static builds.

Concept ACK

does it matter what C++ libarary version is on the system?

FWIW I checked and there's another copy of the C++ library headers under depends/riscv64-unknown-linux-gnu/native/x86_64-linux-musl/include/c++/10.3.0. I think it's supposed to get shared_mutex from there. But also looked at the diff with my local one under /usr/include and there's only a comment difference. So not sure this is the source of the issue.

I've reworked this PR a bit. It's been rebased, and now builds on #20744 (using std::filesystem over boost::filesystem) and #23152 (--enable-lto). bitcoind is no longer linking against any Boost libs, only libevent, which should lessen the potential for build issues. I've also changed the native musl package to point to my fork, where I've added support for GCC 11.2 and binutils 2.37 (will PR upstream soon).

Given that, I've updated the config here so that the toolchain we're now using is GCC 11.2 + binutils 2.37 + musl 1.2.2.

Currently the link invocation for a minimal bitcoind should be (removed duplicate options & shortened lib paths):

# compiler
<path to>/x86_64-linux-musl-g++

# compile options
-m64
-std=c++17
-fdebug-prefix-map=/home/ubuntu/bitcoin=.
-fstack-reuse=none
-Wstack-protector
-fstack-protector-all
-fcf-protection=full
-fstack-clash-protection
-flto
-flto-odr-type-merging
-fPIE
-pipe
-O2
-fno-extended-identifiers
-fvisibility=hidden

# link options
-Wl,--exclude-libs -Wl,ALL
-Wl,-z -Wl,relro
-Wl,-z -Wl,now
-Wl,-z -Wl,separate-code
-pie
-static
-Wl,-O2 
-o bitcoind bitcoind-bitcoind.o init/bitcoind-bitcoind.o

# depends lib search path
-L/home/ubuntu/bitcoin/depends/x86_64-pc-linux-gnu/lib

# internal libs
libbitcoin_common.a
libbitcoin_consensus.a
libbitcoin_server.a
libbitcoin_util.a 

crypto/libbitcoin_crypto_base.a
crypto/libbitcoin_crypto_sse41.a
crypto/libbitcoin_crypto_avx2.a 
crypto/libbitcoin_crypto_shani.a

leveldb/libleveldb.a
leveldb/libmemenv.a

crc32c/libcrc32c.a
crc32c/libcrc32c_sse42.a

secp256k1/.libs/libsecp256k1.a

univalue/.libs/libunivalue.a 

# libstdc++
path/to/libstdc++/in/depends/libstdc++.a

# external libs
-levent_pthreads
-levent
-pthread

@fanquake

Nice work!

It would be interesting to see the output of ldd src/bitcoind :)

@practicalswift

It would be interesting to see the output of ldd src/bitcoind :)

It's:

$ ldd src/bitcoind
	statically linked

It seems like musl-cross-make automagically downloads a bunch of repositories (gcc, etc)... Any way we can "pre-seed" that?

Nice! Playing around with this. Got it to build, great work.

First observation (could be a red herring, take with a grain of salt) is the link line:

x86_64-linux-musl-g++ ... /home/cory/dev/bitcoin2/depends/x86_64-pc-linux-gnu/native/bin/../lib/gcc/x86_64-linux-musl/11.2.0/../../../../x86_64-linux-musl/lib/libstdc++.a libbitcoin_consensus.a crypto/libbitcoin_crypto_base.a crypto/libbitcoin_crypto_sse41.a crypto/libbitcoin_crypto_avx2.a crypto/libbitcoin_crypto_shani.a leveldb/libleveldb.a crc32c/libcrc32c.a crc32c/libcrc32c_sse42.a leveldb/libmemenv.a secp256k1/.libs/libsecp256k1.a ...

That static libstdc++.a seems weird and out of place, I assume libtool is throwing it in there.

If omitted, g++ will add -lstdc++ in the correct place. Note that above it falls before a few c++ libs on the link-line, so I assume it's the added -lstdc++ that's doing the heavy lifting anyway.

I'm mostly mentioning this because the two resulting binaries do differ. Though I haven't looked into how or whether the difference is at all significant.

Rebased this on master now that we are no-longer linking Boost libs, and changed to using GCC 10.3.0, so it, and binutils 2.37 match what we use for Guix builds. Also changed the depends variable to USE_MUSL_LIBC.

Building with -Os, a fully static bitcoind is 8.6mb.

Awesome, concept ACK! Will test soon.

Hah, amazing.

src/bitcoin2 (efcf935) % du -hs ./src/bitcoind
30M     ./src/bitcoind

src/bitcoin2 (efcf935) % ldd ./src/bitcoind
        statically linked

src/bitcoin2 (efcf935) % podman run --rm -it -v ./src/bitcoind:/bitcoind docker.io/library/alpine /bitcoind -version
Bitcoin Core version v22.99.0-efcf9355339f

Concept ACK!

Can this work for clang too?

A couple changes.
Switched to upstream + a patch for binutils 2.37.
Now using musl 1.2.3.
Improved support for cross-compiling.
Explicitly specified Linux headers (currently 4.19.88-1) in config.mak. We can add a patch here and sync up with Guix, likely to 5.15.28 post #25006.

NOTE: you now need to set HOST when doing the depends build. i.e HOST=x86_64-linux-musl.

Can this work for clang too?

We should be able to do something similar with Clang.

Converting this to a draft. I still think using musl libc / this depends based approach is interesting, but will be focussing on #25573 and it's dependant PRs for now.

I will probably maintain this branch going forward, but going to close for now re my comment above.