MSAN Error during connection shutdown #115
Description
Originally posted by @Sjors in bitcoin/bitcoin#30975 (comment)
MSAN says https://cirrus-ci.com/task/6248232848719872:
SUMMARY: MemorySanitizer: use-of-uninitialized-value /usr/src/mp/proxy.cpp:146:25 in mp::Connection::addAsyncCleanup(std::__1::function<void ()>) [10:00:27.923] Exiting
Error happens in ipc_test.cpp:74:
connection_server->onDisconnect([&] { connection_server.reset(); });And stack trace shows mp::Connection being destroyed its RpcSystemBase member being destroyed, which garbage collects mp::ProxyServer objects which call mp::Connection::addAsyncCleanup which accesses invalid memory because mp::Connection object is in the process of being destroyed.
I think this problem points to a general issue with the shutdown sequence and is not just limited to tests because ServeStream function follows the same pattern of destroying the connection object an onDisconnect handler.
If this is true, it means already-destroyed fields of Connection objects are accessed as the objects are being destroyed, as detected by MSAN, but also that some cleanup functions may not run, resulting in small memory leaks.
MSAN Error
[unknown] [src/test/ipc_test.cpp:60] [operator()] LOG0: {IpcPipeTest-15256/b-test-15260} IPC server send response #7 FooInterface.passBlockState$Results (result = (mode = 0, result = 0, reje==15256==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x557fb2ac5ded in std::__1::list<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::__link_nodes(std::__1::__list_node_base<std::__1::function<void ()>, void*>*, std::__1::__list_node_base<std::__1::function<void ()>, void*>*, std::__1::__list_node_base<std::__1::function<void ()>, void*>*) /msan/cxx_build/include/c++/v1/list:957:25
#1 0x557fb2ac5ded in std::__1::__list_iterator<std::__1::function<void ()>, void*> std::__1::list<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::emplace<std::__1::function<void ()>>(std::__1::__list_const_iterator<std::__1::function<void ()>, void*>, std::__1::function<void ()>&&) /msan/cxx_build/include/c++/v1/list:1300:3
#2 0x557fb2ac5ded in mp::Connection::addAsyncCleanup(std::__1::function<void ()>) /usr/src/mp/proxy.cpp:146:25
#3 0x557fb1cf3703 in mp::ProxyServerBase<gen::FooInterface, FooImplementation>::~ProxyServerBase() ci/scratch/build-x86_64-pc-linux-gnu/src/./depends/x86_64-pc-linux-gnu/include/mp/proxy-io.h:480:31
#4 0x557fb1cf25f4 in mp::ProxyServerCustom<gen::FooInterface, FooImplementation>::~ProxyServerCustom() ci/scratch/build-x86_64-pc-linux-gnu/src/./depends/x86_64-pc-linux-gnu/include/mp/proxy.h:160:8
#5 0x557fb1cf25f4 in mp::ProxyServer<gen::FooInterface>::~ProxyServer() ci/scratch/build-x86_64-pc-linux-gnu/src/./ci/scratch/build-x86_64-pc-linux-gnu/src/test/ipc_test.capnp.proxy-types.c++:8:72
#6 0x557fb1cf25f4 in mp::ProxyServer<gen::FooInterface>::~ProxyServer() ci/scratch/build-x86_64-pc-linux-gnu/src/./ci/scratch/build-x86_64-pc-linux-gnu/src/test/ipc_test.capnp.proxy-types.c++:8:48
#7 0x557fb1cf25f4 in mp::ProxyServer<gen::FooInterface>::~ProxyServer() ci/scratch/build-x86_64-pc-linux-gnu/src/./ci/scratch/build-x86_64-pc-linux-gnu/src/test/ipc_test.capnp.proxy-types.c++:8:48
#8 0x557fb1cacea9 in kj::_::HeapDisposer<mp::ProxyServer<gen::FooInterface>>::disposeImpl(void*) const ci/scratch/build-x86_64-pc-linux-gnu/src/./depends/x86_64-pc-linux-gnu/include/kj/memory.h:557:60
#9 0x557fb2b2255b in kj::Disposer::Dispose_<capnp::Capability::Server, true>::dispose(capnp::Capability::Server*, kj::Disposer const&) /usr/src/kj/memory.h:669:14
#10 0x557fb2b2255b in void kj::Disposer::dispose<capnp::Capability::Server>(capnp::Capability::Server*) const /usr/src/kj/memory.h:681:3
#11 0x557fb2b2255b in kj::Own<capnp::Capability::Server, std::nullptr_t>::dispose() /usr/src/kj/memory.h:284:17
#12 0x557fb2b2255b in kj::Own<capnp::Capability::Server, std::nullptr_t>::~Own() /usr/src/kj/memory.h:210:28
#13 0x557fb2b2255b in kj::Maybe<kj::Own<capnp::Capability::Server, std::nullptr_t>>::~Maybe() /usr/src/kj/common.h:1376:7
#14 0x557fb2b2255b in capnp::LocalClient::~LocalClient() /usr/src/capnp/capability.c++:563:3
#15 0x557fb2b22815 in capnp::LocalClient::~LocalClient() /usr/src/capnp/capability.c++:559:34
#16 0x557fb2b22815 in non-virtual thunk to capnp::LocalClient::~LocalClient() /usr/src/capnp/capability.c++
#17 0x557fb2fbaab4 in kj::Refcounted::disposeImpl(void*) const /usr/src/kj/refcount.c++:43:5
#18 0x557fb2c28eff in kj::Disposer::Dispose_<capnp::ClientHook, true>::dispose(capnp::ClientHook*, kj::Disposer const&) /usr/src/kj/memory.h:669:14
#19 0x557fb2c28eff in void kj::Disposer::dispose<capnp::ClientHook>(capnp::ClientHook*) const /usr/src/kj/memory.h:681:3
#20 0x557fb2c28eff in kj::Own<capnp::ClientHook, std::nullptr_t>::dispose() /usr/src/kj/memory.h:284:17
#21 0x557fb2c28eff in kj::Own<capnp::ClientHook, std::nullptr_t>::~Own() /usr/src/kj/memory.h:210:28
#22 0x557fb2c28eff in capnp::Capability::Client::~Client() /usr/src/capnp/capability.h:190:19
#23 0x557fb2c28eff in void kj::dtor<capnp::Capability::Client>(capnp::Capability::Client&) /usr/src/kj/common.h:1060:13
#24 0x557fb2c28eff in kj::_::NullableValue<capnp::Capability::Client>::~NullableValue() /usr/src/kj/common.h:1125:7
#25 0x557fb2c28eff in kj::Maybe<capnp::Capability::Client>::~Maybe() /usr/src/kj/common.h:1376:7
#26 0x557fb2c28eff in capnp::_::RpcSystemBase::Impl::~Impl() /usr/src/capnp/rpc.c++:3567:3
#27 0x557fb2c28bd6 in kj::_::HeapDisposer<capnp::_::RpcSystemBase::Impl>::disposeImpl(void*) const /usr/src/kj/memory.h:557:60
#28 0x557fb2b44d97 in kj::Disposer::Dispose_<capnp::_::RpcSystemBase::Impl, true>::dispose(capnp::_::RpcSystemBase::Impl*, kj::Disposer const&) /usr/src/kj/memory.h:669:14
#29 0x557fb2b44d97 in void kj::Disposer::dispose<capnp::_::RpcSystemBase::Impl>(capnp::_::RpcSystemBase::Impl*) const /usr/src/kj/memory.h:681:3
#30 0x557fb2b44d97 in kj::Own<capnp::_::RpcSystemBase::Impl, std::nullptr_t>::dispose() /usr/src/kj/memory.h:284:17
#31 0x557fb2b44d97 in kj::Own<capnp::_::RpcSystemBase::Impl, std::nullptr_t>::~Own() /usr/src/kj/memory.h:210:28
#32 0x557fb2b44d97 in capnp::_::RpcSystemBase::~RpcSystemBase() /usr/src/capnp/rpc.c++:3678:50
#33 0x557fb2ac42dd in mp::Connection::~Connection() /usr/src/mp/proxy.cpp:105:1
#34 0x557fb1ca1c13 in std::__1::default_delete<mp::Connection>::operator()[abi:de190100](mp::Connection*) const /msan/cxx_build/include/c++/v1/__memory/unique_ptr.h:80:5
#35 0x557fb1ca1c13 in std::__1::unique_ptr<mp::Connection, std::__1::default_delete<mp::Connection>>::reset[abi:de190100](mp::Connection*) /msan/cxx_build/include/c++/v1/__memory/unique_ptr.h:292:7
#36 0x557fb1ca1c13 in IpcPipeTest()::$_0::operator()() const::'lambda0'()::operator()() const ci/scratch/build-x86_64-pc-linux-gnu/src/./src/test/ipc_test.cpp:74:65
#37 0x557fb1ca1c13 in kj::_::Void kj::_::MaybeVoidCaller<kj::_::Void, kj::_::Void>::apply<IpcPipeTest()::$_0::operator()() const::'lambda0'()>(IpcPipeTest()::$_0::operator()() const::'lambda0'()&, kj::_::Void&&) ci/scratch/build-x86_64-pc-linux-gnu/src/./depends/x86_64-pc-linux-gnu/include/kj/async-prelude.h:195:5
#38 0x557fb1ca1c13 in kj::_::TransformPromiseNode<kj::_::Void, kj::_::Void, IpcPipeTest()::$_0::operator()() const::'lambda0'(), kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue&) ci/scratch/build-x86_64-pc-linux-gnu/src/./depends/x86_64-pc-linux-gnu/include/kj/async-inl.h:739:31
#39 0x557fb2dbc3c9 in kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0::operator()() const /usr/src/kj/async.c++:2411:3
#40 0x557fb2dbc3c9 in kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0>(kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0&&) /usr/src/kj/exception.h:371:5
#41 0x557fb2dbc3c9 in kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) /usr/src/kj/async.c++:2411:3
#42 0x557fb2de5fc7 in kj::TaskSet::Task::fire() /usr/src/kj/async.c++:321:11
#43 0x557fb2de7831 in non-virtual thunk to kj::TaskSet::Task::fire() /usr/src/kj/async.c++
#44 0x557fb2db2a28 in kj::EventLoop::turn() /usr/src/kj/async.c++:1806:31
#45 0x557fb2db5bf2 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const /usr/src/kj/async.c++:1970:21
#46 0x557fb2db5bf2 in void kj::WaitScope::runOnStackPool<kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2>(kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2&&) /usr/src/kj/async.h:1368:7
#47 0x557fb2db5bf2 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation) /usr/src/kj/async.c++:1967:17
#48 0x557fb2acfd3c in kj::Promise<unsigned long>::wait(kj::WaitScope&, kj::SourceLocation) /ci_container_base/depends/x86_64-pc-linux-gnu/include/kj/async-inl.h:1357:3
#49 0x557fb2ac764a in mp::EventLoop::loop() /usr/src/mp/proxy.cpp:187:62
#50 0x557fb1c9df16 in IpcPipeTest()::$_0::operator()() const ci/scratch/build-x86_64-pc-linux-gnu/src/./src/test/ipc_test.cpp:75:14
#51 0x557fb1c9df16 in decltype(std::declval<IpcPipeTest()::$_0>()()) std::__1::__invoke[abi:de190100]<IpcPipeTest()::$_0>(IpcPipeTest()::$_0&&) /msan/cxx_build/include/c++/v1/__type_traits/invoke.h:149:25
#52 0x557fb1c9df16 in void std::__1::__thread_execute[abi:de190100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, IpcPipeTest()::$_0>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, IpcPipeTest()::$_0>&, std::__1::__tuple_indices<...>) /msan/cxx_build/include/c++/v1/__thread/thread.h:192:3
#53 0x557fb1c9df16 in void* std::__1::__thread_proxy[abi:de190100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, IpcPipeTest()::$_0>>(void*) /msan/cxx_build/include/c++/v1/__thread/thread.h:201:3
#54 0x7f38033c0a93 (/lib/x86_64-linux-gnu/libc.so.6+0x9ca93) (BuildId: 6d64b17fbac799e68da7ebd9985ddf9b5cb375e6)
#55 0x7f380344dc3b (/lib/x86_64-linux-gnu/libc.so.6+0x129c3b) (BuildId: 6d64b17fbac799e68da7ebd9985ddf9b5cb375e6)
Member fields were destroyed
#0 0x557faee70e5d in __sanitizer_dtor_callback_fields /msan/llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1048:5
#1 0x557fb2ac4097 in std::__1::__list_imp<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::~__list_imp() /msan/cxx_build/include/c++/v1/list:499:15
#2 0x557fb2ac4097 in std::__1::__list_imp<std::__1::function<void ()>, std::__1::allocator<std::__1::function<void ()>>>::~__list_imp() /msan/cxx_build/include/c++/v1/list:619:1
#3 0x557fb2ac4097 in mp::Connection::~Connection() /usr/src/mp/proxy.cpp:105:1
#4 0x557fb1ca1c13 in std::__1::default_delete<mp::Connection>::operator()[abi:de190100](mp::Connection*) const /msan/cxx_build/include/c++/v1/__memory/unique_ptr.h:80:5
#5 0x557fb1ca1c13 in std::__1::unique_ptr<mp::Connection, std::__1::default_delete<mp::Connection>>::reset[abi:de190100](mp::Connection*) /msan/cxx_build/include/c++/v1/__memory/unique_ptr.h:292:7
#6 0x557fb1ca1c13 in IpcPipeTest()::$_0::operator()() const::'lambda0'()::operator()() const ci/scratch/build-x86_64-pc-linux-gnu/src/./src/test/ipc_test.cpp:74:65
#7 0x557fb1ca1c13 in kj::_::Void kj::_::MaybeVoidCaller<kj::_::Void, kj::_::Void>::apply<IpcPipeTest()::$_0::operator()() const::'lambda0'()>(IpcPipeTest()::$_0::operator()() const::'lambda0'()&, kj::_::Void&&) ci/scratch/build-x86_64-pc-linux-gnu/src/./depends/x86_64-pc-linux-gnu/include/kj/async-prelude.h:195:5
#8 0x557fb1ca1c13 in kj::_::TransformPromiseNode<kj::_::Void, kj::_::Void, IpcPipeTest()::$_0::operator()() const::'lambda0'(), kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue&) ci/scratch/build-x86_64-pc-linux-gnu/src/./depends/x86_64-pc-linux-gnu/include/kj/async-inl.h:739:31
#9 0x557fb2dbc3c9 in kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0::operator()() const /usr/src/kj/async.c++:2411:3
#10 0x557fb2dbc3c9 in kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0>(kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)::$_0&&) /usr/src/kj/exception.h:371:5
#11 0x557fb2dbc3c9 in kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&) /usr/src/kj/async.c++:2411:3
#12 0x557fb2de5fc7 in kj::TaskSet::Task::fire() /usr/src/kj/async.c++:321:11
#13 0x557fb2de7831 in non-virtual thunk to kj::TaskSet::Task::fire() /usr/src/kj/async.c++
#14 0x557fb2db2a28 in kj::EventLoop::turn() /usr/src/kj/async.c++:1806:31
#15 0x557fb2db5bf2 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const /usr/src/kj/async.c++:1970:21
#16 0x557fb2db5bf2 in void kj::WaitScope::runOnStackPool<kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2>(kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2&&) /usr/src/kj/async.h:1368:7
#17 0x557fb2db5bf2 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation) /usr/src/kj/async.c++:1967:17
#18 0x557fb2acfd3c in kj::Promise<unsigned long>::wait(kj::WaitScope&, kj::SourceLocation) /ci_container_base/depends/x86_64-pc-linux-gnu/include/kj/async-inl.h:1357:3
#19 0x557fb2ac764a in mp::EventLoop::loop() /usr/src/mp/proxy.cpp:187:62
#20 0x557fb1c9df16 in IpcPipeTest()::$_0::operator()() const ci/scratch/build-x86_64-pc-linux-gnu/src/./src/test/ipc_test.cpp:75:14
#21 0x557fb1c9df16 in decltype(std::declval<IpcPipeTest()::$_0>()()) std::__1::__invoke[abi:de190100]<IpcPipeTest()::$_0>(IpcPipeTest()::$_0&&) /msan/cxx_build/include/c++/v1/__type_traits/invoke.h:149:25
#22 0x557fb1c9df16 in void std::__1::__thread_execute[abi:de190100]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, IpcPipeTest()::$_0>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, IpcPipeTest()::$_0>&, std::__1::__tuple_indices<...>) /msan/cxx_build/include/c++/v1/__thread/thread.h:192:3
#23 0x557fb1c9df16 in void* std::__1::__thread_proxy[abi:de190100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, IpcPipeTest()::$_0>>(void*) /msan/cxx_build/include/c++/v1/__thread/thread.h:201:3
#24 0x7f38033c0a93 (/lib/x86_64-linux-gnu/libc.so.6+0x9ca93) (BuildId: 6d64b17fbac799e68da7ebd9985ddf9b5cb375e6)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /usr/src/mp/proxy.cpp:146:25 in mp::Connection::addAsyncCleanup(std::__1::function<void ()>)