bitcoind hangs waiting for g_requests.empty() #27722
Description
Is there an existing issue for this?
- I have searched the existing issues
Current behaviour
bitcoind hangs
Expected behaviour
bitcoind shouldn't hang
I was able to dump the running threads and it was stuck waiting for g_requests to be empty. The issue seems to be that this callback may not actually be called in case of error:
Lines 222 to 227 in 22139f6
See this issue libevent/libevent#589 for more details. I think what happens is that interrupting bitcoin-cli before it has received the reply calls the error callback and then the success callback is never called.
I think this is also a memory leak since g_requests won't get cleaned up. Whether it's severe or not probably depends on the caller and what type of requests they are making. This new code was introduced here #26742
Steps to reproduce
I was running a simple shell script in 4 terminal windows for fun:
#!/bin/bash
for i in {1..50000}
do
./src/bitcoin-cli -rpcuser=admin -rpcpassword=pass getrawtransaction 217819875434324c412d5e938f7f3c1670fb60814974cf7958f1fac90ee5afb7
done
I was messing around with this script and randomly Ctrl-C'd each terminal window in rapid succession and then when I waited up to a minute and Ctrl-C'd bitcoind, it didn't shutdown. I can reproduce it pretty reliably as well.
Compilation commands:
./autogen.sh; ./configure --without-wallet --with-gui=no --disable-zmq; make -j4
Run-time options (note that I was able to reproduce this with the default number of rpcthreads as well):
./src/bitcoind -txindex -rpcthreads=1 -debug
Output of gcc --version:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
Output of g++ --version:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
My libevent version is 2.1.12
Relevant log output
^C2023-05-22T20:56:10Z [http] Interrupting HTTP server
2023-05-22T20:56:10Z [rpc] Interrupting HTTP RPC server
2023-05-22T20:56:10Z [rpc] Interrupting RPC
2023-05-22T20:56:10Z tor: Thread interrupt
2023-05-22T20:56:10Z Shutdown: In progress...
2023-05-22T20:56:10Z [rpc] Stopping HTTP RPC server
2023-05-22T20:56:10Z torcontrol thread exit
2023-05-22T20:56:10Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-05-22T20:56:10Z addcon thread exit
2023-05-22T20:56:10Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-05-22T20:56:10Z [rpc] Stopping RPC
2023-05-22T20:56:10Z net thread exit
2023-05-22T20:56:10Z [rpc] RPC stopped.
2023-05-22T20:56:10Z [http] Stopping HTTP server
2023-05-22T20:56:10Z [http] Waiting for HTTP worker threads to exit
2023-05-22T20:56:10Z [http] Exited http event loop
2023-05-22T20:56:10Z [http] Waiting for 1 requests to stop HTTP server
The important bit is the very last log line. bitcoind was still alive and periodically printed log lines about feeding dynamic environment data into the RNG.
How did you obtain Bitcoin Core
Compiled from source
What version of Bitcoin Core are you using?
Operating system and version
macOS intel
Machine specifications
No response