bitcoind hits corrupted double-linked list error when running multiple wallet_multiwallet.py tests in parallel #12362
Description
I'm still debugging this, but I'm sharing what I've got so far.
If I run several instances of the wallet_multiwallet.py test in parallel, then bitcoind hits an error, but the test still passes:
./test_runner.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py wallet_multiwallet.py
...*** Error in `/home/ubuntu/bitcoin/src/bitcoind': corrupted double-linked list: 0x0000562c19b928a0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f750c3547e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x82970)[0x7f750c35f970]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f750c361184]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x18)[0x7f750cc53e78]
/home/ubuntu/bitcoin/src/bitcoind(+0x772b9)[0x562c179e92b9]
/home/ubuntu/bitcoin/src/bitcoind(+0x2b8187)[0x562c17c2a187]
/home/ubuntu/bitcoin/src/bitcoind(+0x2aeb36)[0x562c17c20b36]
/home/ubuntu/bitcoin/src/bitcoind(+0x68b2c)[0x562c179dab2c]
/home/ubuntu/bitcoin/src/bitcoind(+0x3f2db)[0x562c179b12db]
/home/ubuntu/bitcoin/src/bitcoind(+0x3222f)[0x562c179a422f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f750c2fd830]
/home/ubuntu/bitcoin/src/bitcoind(+0x3def9)[0x562c179afef9]
======= Memory map: ========
562c17972000-562c17f23000 r-xp 00000000 08:01 516003 /home/ubuntu/bitcoin/src/bitcoind
562c18123000-562c1812e000 r--p 005b1000 08:01 516003 /home/ubuntu/bitcoin/src/bitcoind
562c1812e000-562c18136000 rw-p 005bc000 08:01 516003 /home/ubuntu/bitcoin/src/bitcoind
562c18136000-562c1813c000 rw-p 00000000 00:00 0
562c19a1d000-562c19baf000 rw-p 00000000 00:00 0 [heap]
7f74f8000000-7f74f8023000 rw-p 00000000 00:00 0
7f74f8023000-7f74fc000000 ---p 00000000 00:00 0
7f7500000000-7f7500021000 rw-p 00000000 00:00 0
7f7500021000-7f7504000000 ---p 00000000 00:00 0
7f750613f000-7f7506140000 ---p 00000000 00:00 0
7f7506140000-7f7506940000 rw-p 00000000 00:00 0
7f7506940000-7f7506941000 ---p 00000000 00:00 0
7f7506941000-7f7507141000 rw-p 00000000 00:00 0
7f7507141000-7f7507142000 ---p 00000000 00:00 0
7f7507142000-7f7507942000 rw-p 00000000 00:00 0
7f7507942000-7f7507943000 ---p 00000000 00:00 0
7f7507943000-7f7508143000 rw-p 00000000 00:00 0
7f7508143000-7f7508144000 ---p 00000000 00:00 0
7f7508144000-7f7508944000 rw-p 00000000 00:00 0
7f7508944000-7f7508945000 ---p 00000000 00:00 0
7f7508945000-7f7509145000 rw-p 00000000 00:00 0
7f7509145000-7f7509146000 ---p 00000000 00:00 0
7f7509146000-7f750b948000 rw-p 00000000 00:00 0
7f750ba49000-7f750baa6000 r-xp 00000000 08:01 61325 /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
7f750baa6000-7f750bca5000 ---p 0005d000 08:01 61325 /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
7f750bca5000-7f750bca6000 r--p 0005c000 08:01 61325 /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
7f750bca6000-7f750bca7000 rw-p 0005d000 08:01 61325 /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
7f750bca7000-7f750bcd0000 r-xp 00000000 08:01 61253 /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
7f750bcd0000-7f750becf000 ---p 00029000 08:01 61253 /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
7f750becf000-7f750bed0000 r--p 00028000 08:01 61253 /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
7f750bed0000-7f750bed1000 rw-p 00029000 08:01 61253 /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
7f750bed1000-7f750bed4000 r-xp 00000000 08:01 63277 /lib/x86_64-linux-gnu/libdl-2.23.so
7f750bed4000-7f750c0d3000 ---p 00003000 08:01 63277 /lib/x86_64-linux-gnu/libdl-2.23.so
7f750c0d3000-7f750c0d4000 r--p 00002000 08:01 63277 /lib/x86_64-linux-gnu/libdl-2.23.so
7f750c0d4000-7f750c0d5000 rw-p 00003000 08:01 63277 /lib/x86_64-linux-gnu/libdl-2.23.so
7f750c0d5000-7f750c0dc000 r-xp 00000000 08:01 63311 /lib/x86_64-linux-gnu/librt-2.23.so
7f750c0dc000-7f750c2db000 ---p 00007000 08:01 63311 /lib/x86_64-linux-gnu/librt-2.23.so
7f750c2db000-7f750c2dc000 r--p 00006000 08:01 63311 /lib/x86_64-linux-gnu/librt-2.23.so
7f750c2dc000-7f750c2dd000 rw-p 00007000 08:01 63311 /lib/x86_64-linux-gnu/librt-2.23.so
7f750c2dd000-7f750c49d000 r-xp 00000000 08:01 63272 /lib/x86_64-linux-gnu/libc-2.23.so
7f750c49d000-7f750c69d000 ---p 001c0000 08:01 63272 /lib/x86_64-linux-gnu/libc-2.23.so
7f750c69d000-7f750c6a1000 r--p 001c0000 08:01 63272 /lib/x86_64-linux-gnu/libc-2.23.so
7f750c6a1000-7f750c6a3000 rw-p 001c4000 08:01 63272 /lib/x86_64-linux-gnu/libc-2.23.so
7f750c6a3000-7f750c6a7000 rw-p 00000000 00:00 0
7f750c6a7000-7f750c6bd000 r-xp 00000000 08:01 2039 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f750c6bd000-7f750c8bc000 ---p 00016000 08:01 2039 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f750c8bc000-7f750c8bd000 rw-p 00015000 08:01 2039 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f750c8bd000-7f750c9c5000 r-xp 00000000 08:01 18864 /lib/x86_64-linux-gnu/libm-2.23.so
7f750c9c5000-7f750cbc4000 ---p 00108000 08:01 18864 /lib/x86_64-linux-gnu/libm-2.23.so
7f750cbc4000-7f750cbc5000 r--p 00107000 08:01 18864 /lib/x86_64-linux-gnu/libm-2.23.so
7f750cbc5000-7f750cbc6000 rw-p 00108000 08:01 18864 /lib/x86_64-linux-gnu/libm-2.23.so
7f750cbc6000-7f750cd38000 r-xp 00000000 08:01 8173 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f750cd38000-7f750cf38000 ---p 00172000 08:01 8173 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f750cf38000-7f750cf42000 r--p 00172000 08:01 8173 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f750cf42000-7f750cf44000 rw-p 0017c000 08:01 8173 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f750cf44000-7f750cf48000 rw-p 00000000 00:00 0
7f750cf48000-7f750cfaa000 r-xp 00000000 08:01 62885 /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
7f750cfaa000-7f750d1a9000 ---p 00062000 08:01 62885 /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
7f750d1a9000-7f750d1ad000 r--p 00061000 08:01 62885 /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
7f750d1ad000-7f750d1ae000 rw-p 00065000 08:01 62885 /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
7f750d1ae000-7f750d1f2000 r-xp 00000000 08:01 26071 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
7f750d1f2000-7f750d3f2000 ---p 00044000 08:01 26071 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
7f750d3f2000-7f750d3f3000 r--p 00044000 08:01 26071 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
7f750d3f3000-7f750d3f4000 rw-p 00045000 08:01 26071 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
7f750d3f4000-7f750d3f6000 r-xp 00000000 08:01 61265 /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
7f750d3f6000-7f750d5f5000 ---p 00002000 08:01 61265 /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
7f750d5f5000-7f750d5f6000 r--p 00001000 08:01 61265 /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
7f750d5f6000-7f750d5f7000 rw-p 00002000 08:01 61265 /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
7f750d5f7000-7f750d811000 r-xp 00000000 08:01 28216 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f750d811000-7f750da10000 ---p 0021a000 08:01 28216 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f750da10000-7f750da2c000 r--p 00219000 08:01 28216 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f750da2c000-7f750da38000 rw-p 00235000 08:01 28216 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f750da38000-7f750da3b000 rw-p 00000000 00:00 0
7f750da3b000-7f750da53000 r-xp 00000000 08:01 63271 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f750da53000-7f750dc52000 ---p 00018000 08:01 63271 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f750dc52000-7f750dc53000 r--p 00017000 08:01 63271 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f750dc53000-7f750dc54000 rw-p 00018000 08:01 63271 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f750dc54000-7f750dc58000 rw-p 00000000 00:00 0
7f750dc58000-7f750ddf2000 r-xp 00000000 08:01 552553 /usr/local/BerkeleyDB.4.8/lib/libdb_cxx-4.8.so
7f750ddf2000-7f750dff2000 ---p 0019a000 08:01 552553 /usr/local/BerkeleyDB.4.8/lib/libdb_cxx-4.8.so
7f750dff2000-7f750dff5000 r--p 0019a000 08:01 552553 /usr/local/BerkeleyDB.4.8/lib/libdb_cxx-4.8.so
7f750dff5000-7f750dff8000 rw-p 0019d000 08:01 552553 /usr/local/BerkeleyDB.4.8/lib/libdb_cxx-4.8.so
7f750dff8000-7f750dffe000 r-xp 00000000 08:01 60749 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0
7f750dffe000-7f750e1fe000 ---p 00006000 08:01 60749 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0
7f750e1fe000-7f750e1ff000 r--p 00006000 08:01 60749 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0
7f750e1ff000-7f750e200000 rw-p 00007000 08:01 60749 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0
7f750e200000-7f750e224000 r-xp 00000000 08:01 60765 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
7f750e224000-7f750e423000 ---p 00024000 08:01 60765 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
7f750e423000-7f750e425000 r--p 00023000 08:01 60765 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
7f750e425000-7f750e426000 rw-p 00025000 08:01 60765 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
7f750e426000-7f750e4a0000 r-xp 00000000 08:01 60742 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0
7f750e4a0000-7f750e6a0000 ---p 0007a000 08:01 60742 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0
7f750e6a0000-7f750e6a3000 r--p 0007a000 08:01 60742 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0
7f750e6a3000-7f750e6a4000 rw-p 0007d000 08:01 60742 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0
7f750e6a4000-7f750e6bb000 r-xp 00000000 08:01 60843 /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0
7f750e6bb000-7f750e8ba000 ---p 00017000 08:01 60843 /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0
7f750e8ba000-7f750e8bb000 r--p 00016000 08:01 60843 /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0
7f750e8bb000-7f750e8bc000 rw-p 00017000 08:01 60843 /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0
7f750e8bc000-7f750e8bf000 r-xp 00000000 08:01 60743 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0
7f750e8bf000-7f750eabe000 ---p 00003000 08:01 60743 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0
7f750eabe000-7f750eabf000 r--p 00002000 08:01 60743 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0
7f750eabf000-7f750eac0000 rw-p 00003000 08:01 60743 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0
7f750eac0000-7f750eae6000 r-xp 00000000 08:01 19651 /lib/x86_64-linux-gnu/ld-2.23.so
7f750eb1f000-7f750eb2f000 rw-p 00000000 00:00 0
7f750eb2f000-7f750ecc7000 r--p 00000000 08:01 7864 /usr/lib/locale/locale-archive
7f750ecc7000-7f750ecd6000 rw-p 00000000 00:00 0
7f750ecdd000-7f750ecde000 rw-p 00000000 00:00 0
7f750ecde000-7f750ece5000 r--s 00000000 08:01 63406 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f750ece5000-7f750ece6000 r--p 00025000 08:01 19651 /lib/x86_64-linux-gnu/ld-2.23.so
7f750ece6000-7f750ece7000 rw-p 00026000 08:01 19651 /lib/x86_64-linux-gnu/ld-2.23.so
7f750ece7000-7f750ece8000 rw-p 00000000 00:00 0
7ffd2c413000-7ffd2c434000 rw-p 00000000 00:00 0 [stack]
7ffd2c475000-7ffd2c477000 r--p 00000000 00:00 0 [vvar]
7ffd2c477000-7ffd2c479000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
wallet_multiwallet.py passed, Duration: 5 s
[...]
I can only see this output if I run through the test_runner, not if I run the test directly (this might just be something to do with where the test_framework is sending stderr).
I've bisected this to somewhere between commit 886a92f and 9bb59cf . Note that prior to 90600bc, this test was called multiwallet.py.
This test is testing a lot of cases of bitcoind failing to start because of bad -wallet parameters being passed in, so the test is expecting the process to exit. However, we obviously shouldn't be hitting memory corruption like I see in the output above.
No-one else here has been able to reproduce this yet, but I can hit it reliably on master and v0.16.0rc2. One guess of somewhere to look is bdb since this is being tickled by the wallet code.