ci: Use TSan new runtime (llvm-16, take 3) #27298

maflcko · 2023-03-22T11:52:08Z

The previous two attempts failed:

llvm-14: Failed in ci: Temporarily use clang-13 to work around clang-14 TSan bug #24572
llvm-15: Failed in ci: Revert tsan task changes #26775

However, now that the bug is known and fixed, it should be good to go. See also #26775 (comment)

DrahtBot · 2023-03-22T11:52:11Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	fanquake
Concept ACK	dergoegge

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#27340 (ci: Use Cirrus CI dockerfile env by MarcoFalke)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

fanquake · 2023-03-22T13:21:46Z

Imagine how much nicer life would be if we could upgrade the sanitizer infrastructure for the bitcoin consensus code without having to make some random gui sub dependency compile with a newer version of Clang. I'll take a look. Maybe we can just drop a patch into depends.

        if (os2->achVendID && os2->achVendID[0] != 0)
            ~~~~~^~~~~~~~~ ~~
  CC       fcname.lo
  CC       fcobjs.lo
  CC       fcpat.lo
  CC       fcrange.lo
  CC       fcserialize.lo
  CC       fcstat.lo
  CC       fcstr.lo
  CC       fcweight.lo
  CC       fcxml.lo
  CC       ftglue.lo
1 warning generated.
  CC       fcarch.o
  CCLD     fcarch
  CCLD     libfontconfig.la
make[4]: Leaving directory '/tmp/cirrus-ci-build/depends/work/build/x86_64-pc-linux-gnu/fontconfig/2.12.6-6d71f6a3836/src'
make[3]: Leaving directory '/tmp/cirrus-ci-build/depends/work/build/x86_64-pc-linux-gnu/fontconfig/2.12.6-6d71f6a3836/src'
Making all in fc-cache
make[3]: Entering directory '/tmp/cirrus-ci-build/depends/work/build/x86_64-pc-linux-gnu/fontconfig/2.12.6-6d71f6a3836/fc-cache'
  CC       fc-cache.o
  CCLD     fc-cache
make[3]: Leaving directory '/tmp/cirrus-ci-build/depends/work/build/x86_64-pc-linux-gnu/fontconfig/2.12.6-6d71f6a3836/fc-cache'
Making all in fc-cat
make[3]: Entering directory '/tmp/cirrus-ci-build/depends/work/build/x86_64-pc-linux-gnu/fontconfig/2.12.6-6d71f6a3836/fc-cat'
  CC       fc-cat.o
fc-cat.c:252:10: warning: variable 'ret' set but not used [-Wunused-but-set-variable]
    int         ret = 0;
                ^
1 warning generated.
  CCLD     fc-cat
make[3]: Leaving directory '/tmp/cirrus-ci-build/depends/work/build/x86_64-pc-linux-gnu/fontconfig/2.12.6-6d71f6a3836/fc-cat'
Making all in fc-list
make[3]: Entering directory '/tmp/cirrus-ci-build/depends/work/build/x86_64-pc-linux-gnu/fontconfig/2.12.6-6d71f6a3836/fc-list'
  CC       fc-list.o
fc-list.c:119:27: error: call to undeclared function 'strdup'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
            format = (FcChar8 *) strdup (optarg);
                                 ^
fc-list.c:119:27: note: did you mean 'strcmp'?
/usr/include/string.h:156:12: note: 'strcmp' declared here
extern int strcmp (const char *__s1, const char *__s2)
           ^
fc-list.c:119:15: warning: cast to 'FcChar8 *' (aka 'unsigned char *') from smaller integer type 'int' [-Wint-to-pointer-cast]
            format = (FcChar8 *) strdup (optarg);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning and 1 error generated.
make[3]: *** [Makefile:510: fc-list.o] Error 1

dergoegge · 2023-03-22T13:52:31Z

Concept ACK

dergoegge · 2023-03-22T14:05:51Z

Could we drop the GUI from the TSan task?

maflcko · 2023-03-22T14:24:04Z

Could we drop the GUI from the TSan task?

Yes, but there is another (linker) bug.

fanquake · 2023-03-22T18:00:11Z

Could we drop the GUI from the TSan task?

Concept ACK.

Yes, but there is another (linker) bug.

Looks like just explicitly installing libclang-rt-16-dev will solve this. This branch works for me, for running the TSAN CI job on x86_64: https://github.com/fanquake/bitcoin/tree/27298_plus_libclang:

ALL                                                    | ✓ Passed  | 3493 s (accumulated) 
Runtime: 991 s

Stop and remove CI container by ID
48aee56b02c33313b240f21c0b9924c68bfa9fcaaa16d0557adcd31b1efb73be

real	32m28.067s

Still testing on aarch64.

fanquake · 2023-03-22T18:11:42Z

Ran into a failure (with the above branch, and NO_BDB=1), on aarch64:

2023-03-22T18:02:11.462544Z (mocktime: 2020-08-31T15:34:12Z) [txindex] [util/thread.cpp:20] [TraceThread] txindex thread start
test/txindex_tests.cpp(38): fatal error: in "txindex_tests/txindex_initial_sync": critical check time_start + timeout_ms > GetTimeMillis() has failed
LLVMSymbolizer: error reading file: No such file or directory
make[3]: Leaving directory '/home/fedora/ci_scratch/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src'
make[2]: *** [Makefile:19828: check-am] Error 2
make[2]: Leaving directory '/home/fedora/ci_scratch/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src'
make[1]: *** [Makefile:19493: check-recursive] Error 1
make[1]: Leaving directory '/home/fedora/ci_scratch/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src'
make: *** [Makefile:816: check-recursive] Error 1
==================
WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=32811)
  Write of size 8 at 0xffffd3434508 by main thread:
    #0 BaseIndex::~BaseIndex() src/index/base.cpp:80:1 (test_bitcoin+0xc540e4) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #1 TxIndex::~TxIndex() src/index/txindex.cpp:56:19 (test_bitcoin+0xc6cf08) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #2 txindex_tests::txindex_initial_sync::test_method() src/test/txindex_tests.cpp:82:1 (test_bitcoin+0x7c2450) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #3 txindex_tests::txindex_initial_sync_invoker() src/test/txindex_tests.cpp:17:1 (test_bitcoin+0x7c0bac) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #4 boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:117:11 (test_bitcoin+0x2b7868) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #5 boost::function0<void>::operator()() const /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:763:14 (test_bitcoin+0x24f2dc) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #6 boost::detail::forward::operator()() /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1388:32 (test_bitcoin+0x24f2dc)
    #7 boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:137:18 (test_bitcoin+0x24f2dc)
    #8 boost::function0<int>::operator()() const /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:763:14 (test_bitcoin+0x1e3304) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #9 int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:301:30 (test_bitcoin+0x1e3304)
    #10 boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:903:16 (test_bitcoin+0x1e3304)
    #11 boost::execution_monitor::execute(boost::function<int ()> const&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1301:16 (test_bitcoin+0x1e35fc) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #12 boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1397:5 (test_bitcoin+0x1dce40) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #13 boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x1dce40)
    #14 boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x20cd7c) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #15 boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x20d0a8) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #16 boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x20d0a8) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #17 boost::unit_test::framework::run(unsigned long, bool) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x1dbde4) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #18 boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x1f5748) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #19 main /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x1f5cf4) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)

  Previous read of size 8 at 0xffffd3434508 by thread T4:
    #0 BaseIndex::ThreadSync() src/index/base.cpp:217:18 (test_bitcoin+0xc54fbc) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #1 BaseIndex::Start()::$_0::operator()() const src/index/base.cpp:404:73 (test_bitcoin+0xc58740) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #2 decltype(std::declval<BaseIndex::Start()::$_0&>()()) std::__1::__invoke[abi:v160000]<BaseIndex::Start()::$_0&>(BaseIndex::Start()::$_0&) /usr/lib/llvm-16/bin/../include/c++/v1/__functional/invoke.h:394:23 (test_bitcoin+0xc58740)
    #3 void std::__1::__invoke_void_return_wrapper<void, true>::__call<BaseIndex::Start()::$_0&>(BaseIndex::Start()::$_0&) /usr/lib/llvm-16/bin/../include/c++/v1/__functional/invoke.h:487:9 (test_bitcoin+0xc58740)
    #4 std::__1::__function::__alloc_func<BaseIndex::Start()::$_0, std::__1::allocator<BaseIndex::Start()::$_0>, void ()>::operator()[abi:v160000]() /usr/lib/llvm-16/bin/../include/c++/v1/__functional/function.h:185:16 (test_bitcoin+0xc58740)
    #5 std::__1::__function::__func<BaseIndex::Start()::$_0, std::__1::allocator<BaseIndex::Start()::$_0>, void ()>::operator()() /usr/lib/llvm-16/bin/../include/c++/v1/__functional/function.h:356:12 (test_bitcoin+0xc58740)
    #6 std::__1::__function::__value_func<void ()>::operator()[abi:v160000]() const /usr/lib/llvm-16/bin/../include/c++/v1/__functional/function.h:510:16 (test_bitcoin+0x10ca298) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #7 std::__1::function<void ()>::operator()() const /usr/lib/llvm-16/bin/../include/c++/v1/__functional/function.h:1156:12 (test_bitcoin+0x10ca298)
    #8 util::TraceThread(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>) src/util/thread.cpp:21:9 (test_bitcoin+0x10ca298)
    #9 decltype(std::declval<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>)>()(std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(), std::declval<BaseIndex::Start()::$_0>())) std::__1::__invoke[abi:v160000]<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, BaseIndex::Start()::$_0>(void (*&&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&, BaseIndex::Start()::$_0&&) /usr/lib/llvm-16/bin/../include/c++/v1/__functional/invoke.h:394:23 (test_bitcoin+0xc5828c) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #10 void std::__1::__thread_execute[abi:v160000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, BaseIndex::Start()::$_0, 2ul, 3ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, BaseIndex::Start()::$_0>&, std::__1::__tuple_indices<2ul, 3ul>) /usr/lib/llvm-16/bin/../include/c++/v1/thread:282:5 (test_bitcoin+0xc5828c)
    #11 void* std::__1::__thread_proxy[abi:v160000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, BaseIndex::Start()::$_0>>(void*) /usr/lib/llvm-16/bin/../include/c++/v1/thread:293:5 (test_bitcoin+0xc5828c)

  Location is stack of main thread.

  Location is global '??' at 0xffffd3417000 ([stack]+0x1d508)

  Thread T4 'b-txindex' (tid=32816, running) created by main thread at:
    #0 pthread_create <null> (test_bitcoin+0x1376ec) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #1 std::__1::__libcpp_thread_create[abi:v160000](unsigned long*, void* (*)(void*), void*) /usr/lib/llvm-16/bin/../include/c++/v1/__threading_support:378:10 (test_bitcoin+0xc57f1c) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #2 std::__1::thread::thread<void (*)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, BaseIndex::Start()::$_0, void>(void (*&&)(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, BaseIndex::Start()::$_0&&) /usr/lib/llvm-16/bin/../include/c++/v1/thread:309:16 (test_bitcoin+0xc57f1c)
    #3 BaseIndex::Start() src/index/base.cpp:404:21 (test_bitcoin+0xc57f1c)
    #4 txindex_tests::txindex_initial_sync::test_method() src/test/txindex_tests.cpp:32:5 (test_bitcoin+0x7c150c) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #5 txindex_tests::txindex_initial_sync_invoker() src/test/txindex_tests.cpp:17:1 (test_bitcoin+0x7c0bac) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #6 boost::detail::function::void_function_invoker0<void (*)(), void>::invoke(boost::detail::function::function_buffer&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:117:11 (test_bitcoin+0x2b7868) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #7 boost::function0<void>::operator()() const /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:763:14 (test_bitcoin+0x24f2dc) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #8 boost::detail::forward::operator()() /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1388:32 (test_bitcoin+0x24f2dc)
    #9 boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:137:18 (test_bitcoin+0x24f2dc)
    #10 boost::function0<int>::operator()() const /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/function/function_template.hpp:763:14 (test_bitcoin+0x1e3304) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #11 int boost::detail::do_invoke<boost::shared_ptr<boost::detail::translator_holder_base>, boost::function<int ()>>(boost::shared_ptr<boost::detail::translator_holder_base> const&, boost::function<int ()> const&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:301:30 (test_bitcoin+0x1e3304)
    #12 boost::execution_monitor::catch_signals(boost::function<int ()> const&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:903:16 (test_bitcoin+0x1e3304)
    #13 boost::execution_monitor::execute(boost::function<int ()> const&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1301:16 (test_bitcoin+0x1e35fc) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #14 boost::execution_monitor::vexecute(boost::function<void ()> const&) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/execution_monitor.ipp:1397:5 (test_bitcoin+0x1dce40) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #15 boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_monitor.ipp:49:9 (test_bitcoin+0x1dce40)
    #16 boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:815:44 (test_bitcoin+0x20cd7c) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #17 boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x20d0a8) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #18 boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:784:58 (test_bitcoin+0x20d0a8) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #19 boost::unit_test::framework::run(unsigned long, bool) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/framework.ipp:1722:29 (test_bitcoin+0x1dbde4) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #20 boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_main.ipp:250:9 (test_bitcoin+0x1f5748) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)
    #21 main /home/fedora/ci_scratch/depends/aarch64-unknown-linux-gnu/include/boost/test/impl/unit_test_main.ipp:306:12 (test_bitcoin+0x1f5cf4) (BuildId: b7b04616b4944ef4985d18b976ee210f40ecea99)

SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) src/index/base.cpp:80:1 in BaseIndex::~BaseIndex()
==================

real	28m15.335s

Full log: https://gist.github.com/fanquake/f5b4a6e114b727a290bf078e3b99990c.

maflcko · 2023-03-24T14:16:41Z

Ran into a failure (with the above branch, and NO_BDB=1), on aarch64:

Does this happen on master as well?

maflcko · 2023-03-24T14:19:31Z

See also #26188

fanquake · 2023-03-24T17:06:02Z

Does this happen on master as well?

A different failure with master (e352f5a):

make  check-TESTS check-local
make[3]: Entering directory '/home/fedora/bitcoin/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src'
Running tests: addrman_tests from test/addrman_tests.cpp
Running tests: allocator_tests from test/allocator_tests.cpp
Running tests: amount_tests from test/amount_tests.cpp
make[4]: Entering directory '/home/fedora/bitcoin/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src'
/bin/bash: line 2: 27679 Segmentation fault      (core dumped) test/test_bitcoin --catch_system_errors=no -l test_suite -t "$( cat test/amount_tests.cpp | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1 )" -- DEBUG_LOG_OUT > "$TEST_LOGFILE" 2>&1
ThreadSanitizer: CHECK failed: tsan_platform_linux.cpp:310 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff) (tid=27679)
/bin/bash: line 2: 27670 Segmentation fault      (core dumped) test/test_bitcoin --catch_system_errors=no -l test_suite -t "$( cat test/addrman_tests.cpp | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1 )" -- DEBUG_LOG_OUT > "$TEST_LOGFILE" 2>&1
make[3]: *** [Makefile:21823: test/amount_tests.cpp.test] Error 1
make[3]: *** Waiting for unfinished jobs....
/bin/bash: line 2: 27677 Segmentation fault      (core dumped) test/test_bitcoin --catch_system_errors=no -l test_suite -t "$( cat test/allocator_tests.cpp | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1 )" -- DEBUG_LOG_OUT > "$TEST_LOGFILE" 2>&1
ThreadSanitizer: CHECK failed: tsan_platform_linux.cpp:310 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff) (tid=27670)
make[3]: *** [Makefile:21823: test/addrman_tests.cpp.test] Error 1
../build-aux/test-driver: line 112: 27688 Segmentation fault      (core dumped) "$@" >> "$log_file" 2>&1
FAIL: minisketch/test
ThreadSanitizer: CHECK failed: tsan_platform_linux.cpp:310 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff) (tid=27677)
make[3]: *** [Makefile:21823: test/allocator_tests.cpp.test] Error 1
../build-aux/test-driver: line 112: 27710 Segmentation fault      (core dumped) "$@" >> "$log_file" 2>&1
FAIL: univalue/test/object
../build-aux/test-driver: line 112: 27709 Segmentation fault      (core dumped) "$@" >> "$log_file" 2>&1
FAIL: univalue/test/unitester
==============================================
   Bitcoin Core 24.99.0: src/test-suite.log
==============================================

FAIL: minisketch/test
=====================

ThreadSanitizer: CHECK failed: tsan_platform_linux.cpp:310 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff) (tid=27688)
FAIL minisketch/test (exit status: 139)

FAIL: univalue/test/object
==========================

ThreadSanitizer: CHECK failed: tsan_platform_linux.cpp:310 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff) (tid=27710)
FAIL univalue/test/object (exit status: 139)

FAIL: univalue/test/unitester
=============================

ThreadSanitizer: CHECK failed: tsan_platform_linux.cpp:310 "((personality(old_personality | ADDR_NO_RANDOMIZE))) != ((-1))" (0xffffffffffffffff, 0xffffffffffffffff) (tid=27709)
FAIL univalue/test/unitester (exit status: 139)

fanquake · 2023-03-27T14:41:03Z

Could rebase on #27301, and see what happens when you drop 6a05c84 ?

fanquake · 2023-03-27T15:57:06Z

Looks like the explicit libclang-rt-dev package installation was lost, so configure is failing.

9cbc1c2 depends: make fontconfig build under clang-16 (fanquake) Pull request description: Use the same workaround we've applied to qrencode, and other packages. Fontconfig not building is currently a blocker for fuzz/sanitizer infra upgrades (#27298). For now, this is also more straightforward than bumping the package, which introduces more complexity/usage of gperf. Closes: #27299. ACKs for top commit: hebasto: ACK 9cbc1c2 Tree-SHA512: 387ea1a73e3429f166ef5278305a56cb3c69b6e3fc8a21a66521738e313e3fe783f042759b396cd88e28c10918a4427fb836a8dfecc5a846723b6f6c6a7ade51

9cbc1c2 depends: make fontconfig build under clang-16 (fanquake) Pull request description: Use the same workaround we've applied to qrencode, and other packages. Fontconfig not building is currently a blocker for fuzz/sanitizer infra upgrades (bitcoin#27298). For now, this is also more straightforward than bumping the package, which introduces more complexity/usage of gperf. Closes: bitcoin#27299. ACKs for top commit: hebasto: ACK 9cbc1c2 Tree-SHA512: 387ea1a73e3429f166ef5278305a56cb3c69b6e3fc8a21a66521738e313e3fe783f042759b396cd88e28c10918a4427fb836a8dfecc5a846723b6f6c6a7ade51

fanquake

ACK faf4aca - I still see this failure on aarch64, but that isn't really a regression, as running this tests was already broken for me. I'll open a separate issue, and we can follow up.

faf4aca ci: Use TSan new runtime (llvm-16, take 3) (MarcoFalke) Pull request description: The previous two attempts failed: * llvm-14: Failed in bitcoin#24572 * llvm-15: Failed in bitcoin#26775 However, now that the bug is known and fixed, it should be good to go. See also bitcoin#26775 (comment) ACKs for top commit: fanquake: ACK faf4aca - I still see [this](bitcoin#27298 (comment)) failure on aarch64, but that isn't really a regression, as running this tests was already broken for me. I'll open a separate issue, and we can follow up. Tree-SHA512: 372b53c4d42ca7f527dae4a2b5bc5ab33c816930daf7a3479d20ea7749159a0b19cfd8d76244b95b03130e4a3d12ddbbb74668b8f7e9fc272cf1084f53b7ff9b

Similar to bitcoin#27298.

a634c28 ci: use LLVM/clang-16 in native_fuzz (ASAN) job (fanquake) Pull request description: Similar to #27298. ACKs for top commit: dergoegge: utACK a634c28 Tree-SHA512: 7a2625a3ac83710063d941dcbca42431b3b79a1380872fd2c566c0ab3041d8123d7dcddeb8a4972efd0ef6496b15bbe0b39b6d2de84df81fcdd8d68e1248fbc5

Similar to bitcoin#27298.

f952e67 ci: remove usage of untrusted bpfcc-tools (fanquake) 1232c2f ci: use LLVM/clang-16 in native_asan job (fanquake) Pull request description: Similar to #27298. Working for me on `x86_64` and solves the issue I currently see with TSAN on `aarch64` with master (6882828): ```bash crc32c/src/crc32c_arm64.cc:101:26: runtime error: load of misaligned address 0xffff84400406 for type 'uint64_t' (aka 'unsigned long'), which requires 8 byte alignment 0xffff84400406: note: pointer points here b9 c5 22 00 01 01 1a 6c 65 76 65 6c 64 62 2e 42 79 74 65 77 69 73 65 43 6f 6d 70 61 72 61 74 6f ^ #0 0xaaaaaddaf0b4 in crc32c::ExtendArm64(unsigned int, unsigned char const*, unsigned long) src/./src/crc32c/src/crc32c_arm64.cc:101:26 #1 0xaaaaadd2c838 in leveldb::crc32c::Value(char const*, unsigned long) src/./leveldb/util/crc32c.h:20:60 #2 0xaaaaadd2c838 in leveldb::log::Reader::ReadPhysicalRecord(leveldb::Slice*) src/./src/leveldb/db/log_reader.cc:246:29 #3 0xaaaaadd2ba9c in leveldb::log::Reader::ReadRecord(leveldb::Slice*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) src/./src/leveldb/db/log_reader.cc:72:38 #4 0xaaaaadd41710 in leveldb::VersionSet::Recover(bool*) src/./src/leveldb/db/version_set.cc:910:19 #5 0xaaaaadcf9fec in leveldb::DBImpl::Recover(leveldb::VersionEdit*, bool*) src/./src/leveldb/db/db_impl.cc:320:18 #6 0xaaaaadd12068 in leveldb::DB::Open(leveldb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, leveldb::DB**) src/./src/leveldb/db/db_impl.cc:1487:20 #7 0xaaaaad314e80 in CDBWrapper::CDBWrapper(DBParams const&) src/./src/dbwrapper.cpp:156:30 #8 0xaaaaace94880 in CBlockTreeDB::CBlockTreeDB(DBParams const&) src/./txdb.h:89:23 #9 0xaaaaace94880 in std::_MakeUniq<CBlockTreeDB>::__single_object std::make_unique<CBlockTreeDB, DBParams>(DBParams&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 #10 0xaaaaace94880 in ChainTestingSetup::ChainTestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) src/./src/test/util/setup_common.cpp:188:51 #11 0xaaaaace95da0 in TestingSetup::TestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&, bool, bool) src/./src/test/util/setup_common.cpp:243:7 #12 0xaaaaace96730 in TestChain100Setup::TestChain100Setup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&, bool, bool) src/./src/test/util/setup_common.cpp:274:7 #13 0xaaaaac1ddbc8 in blockfilter_index_tests::BuildChainTestingSetup::BuildChainTestingSetup() src/./src/test/blockfilter_index_tests.cpp:26:8 #14 0xaaaaac1ddbc8 in blockfilter_index_tests::blockfilter_index_initial_sync::blockfilter_index_initial_sync() src/./src/test/blockfilter_index_tests.cpp:112:1 #15 0xaaaaac1ddbc8 in blockfilter_index_tests::blockfilter_index_initial_sync_invoker() src/./src/test/blockfilter_index_tests.cpp:112:1 #16 0xaaaaabf08f7c in boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:763:14 #17 0xaaaaabf95468 in boost::detail::forward::operator()() /usr/include/boost/test/impl/execution_monitor.ipp:1388:32 #18 0xaaaaabf95468 in boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:137:18 #19 0xaaaaabf8e12c in boost::function0<int>::operator()() const /usr/include/boost/function/function_template.hpp:763:14 #20 0xaaaaabe7be14 in boost::execution_monitor::catch_signals(boost::function<int ()> const&) /usr/include/boost/test/impl/execution_monitor.ipp:903:16 #21 0xaaaaabe7c1c0 in boost::execution_monitor::execute(boost::function<int ()> const&) /usr/include/boost/test/impl/execution_monitor.ipp:1301:16 #22 0xaaaaabe6f47c in boost::execution_monitor::vexecute(boost::function<void ()> const&) /usr/include/boost/test/impl/execution_monitor.ipp:1397:5 #23 0xaaaaabe75124 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) /usr/include/boost/test/impl/unit_test_monitor.ipp:49:9 #24 0xaaaaabed19fc in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /usr/include/boost/test/impl/framework.ipp:815:44 #25 0xaaaaabed0f6c in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /usr/include/boost/test/impl/framework.ipp:784:58 #26 0xaaaaabed0f6c in boost::unit_test::framework::state::execute_test_tree(unsigned long, unsigned long, boost::unit_test::framework::state::random_generator_helper const*) /usr/include/boost/test/impl/framework.ipp:784:58 #27 0xaaaaabe73878 in boost::unit_test::framework::run(unsigned long, bool) /usr/include/boost/test/impl/framework.ipp:1721:29 #28 0xaaaaabe9d244 in boost::unit_test::unit_test_main(boost::unit_test::test_suite* (*)(int, char**), int, char**) /usr/include/boost/test/impl/unit_test_main.ipp:250:9 #29 0xffff8f0773f8 (/lib/aarch64-linux-gnu/libc.so.6+0x273f8) (BuildId: f37f3aa07c797e333fd106472898d361f71798f5) #30 0xffff8f0774c8 in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x274c8) (BuildId: f37f3aa07c797e333fd106472898d361f71798f5) #31 0xaaaaabda55ac in _start (/home/fedora/ci_scratch/ci/scratch/build/bitcoin-aarch64-unknown-linux-gnu/src/test/test_bitcoin+0x10e55ac) (BuildId: b7909adaefd9db6cd6a7c4d3d40207cf6bdaf4b3) SUMMARY: UndefinedBehaviorSanitizer: misaligned-pointer-use crc32c/src/crc32c_arm64.cc:101:26 in ``` ACKs for top commit: dergoegge: utACK f952e67 MarcoFalke: lgtm ACK f952e67 Tree-SHA512: 9dee2abf73d3f23bb9979bfb453b48e39f0b7a5f58d43824ecf053a53e9800ed413b915382b274d1a84baf2999683e3b485463e377e0455b3f0ead65ed1d1916

Similar to bitcoin#27298.

faf4aca ci: Use TSan new runtime (llvm-16, take 3) (MarcoFalke) Pull request description: The previous two attempts failed: * llvm-14: Failed in bitcoin#24572 * llvm-15: Failed in bitcoin#26775 However, now that the bug is known and fixed, it should be good to go. See also bitcoin#26775 (comment) ACKs for top commit: fanquake: ACK faf4aca - I still see [this](bitcoin#27298 (comment)) failure on aarch64, but that isn't really a regression, as running this tests was already broken for me. I'll open a separate issue, and we can follow up. Tree-SHA512: 372b53c4d42ca7f527dae4a2b5bc5ab33c816930daf7a3479d20ea7749159a0b19cfd8d76244b95b03130e4a3d12ddbbb74668b8f7e9fc272cf1084f53b7ff9b

backport: bitcoin#27298 (use clang 16)

DrahtBot added the Tests label Mar 22, 2023

maflcko changed the title ~~ci: Use TSan v2 (llvm-16, take 3)~~ ci: Use TSan new runtime (llvm-16, take 3) Mar 22, 2023

fanquake mentioned this pull request Mar 22, 2023

depends: make fontconfig build under clang-16 #27301

Merged

maflcko force-pushed the 2303-ci-tsan-v2-llvm-16-take-3- branch from 00cf7c9 to 6a05c84 Compare March 22, 2023 14:25

maflcko mentioned this pull request Mar 24, 2023

ci: Add missing libclang-rt-dev package #27323

Closed

DrahtBot mentioned this pull request Mar 27, 2023

ci: Use Cirrus CI dockerfile env #27340

Merged

maflcko force-pushed the 2303-ci-tsan-v2-llvm-16-take-3- branch from 6a05c84 to 28f406b Compare March 27, 2023 14:44

DrahtBot added the Needs rebase label Mar 27, 2023

ci: Use TSan new runtime (llvm-16, take 3)

faf4aca

maflcko force-pushed the 2303-ci-tsan-v2-llvm-16-take-3- branch from 28f406b to faf4aca Compare March 28, 2023 08:02

DrahtBot removed the Needs rebase label Mar 28, 2023

maflcko marked this pull request as ready for review March 28, 2023 09:50

fanquake approved these changes Mar 28, 2023

View reviewed changes

fanquake merged commit 2200086 into bitcoin:master Mar 28, 2023

maflcko deleted the 2303-ci-tsan-v2-llvm-16-take-3-🌡 branch March 28, 2023 13:47

fanquake mentioned this pull request Mar 28, 2023

index: ThreadSanitizer: data race on vptr #27355

Closed

fanquake added a commit to fanquake/bitcoin that referenced this pull request Mar 29, 2023

ci: use clang/llvm-16 in asan job

fd0a58a

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Mar 29, 2023

ci: use LLVM/clang-16 in ASAN job

a341c1d

Similar to bitcoin#27298.

fanquake mentioned this pull request Mar 29, 2023

ci: use LLVM/clang-16 in native_asan job #27360

Merged

fanquake added a commit to fanquake/bitcoin that referenced this pull request Mar 29, 2023

ci: use LLVM/clang-16 in ASAN jobs

1188055

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Mar 29, 2023

ci: use LLVM/clang-16 in native_asan job

97edee3

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Mar 29, 2023

ci: use LLVM/clang-16 in native_fuzz (ASAN) job

a634c28

Similar to bitcoin#27298.

fanquake mentioned this pull request Mar 29, 2023

ci: use LLVM/clang-16 in native_fuzz (ASAN) job #27363

Merged

fanquake added a commit to fanquake/bitcoin that referenced this pull request Mar 31, 2023

ci: use LLVM/clang-16 in native_asan job

579e027

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 6, 2023

ci: use LLVM/clang-16 in native_asan job

1759d10

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 17, 2023

ci: use LLVM/clang-16 in native_asan job

0af85aa

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 18, 2023

ci: use LLVM/clang-16 in native_asan job

c2f757f

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 20, 2023

ci: use LLVM/clang-16 in native_asan job

4cd3b39

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 20, 2023

ci: use LLVM/clang-16 in native_asan job

35362a1

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 21, 2023

ci: use LLVM/clang-16 in native_asan job

987567f

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 21, 2023

ci: use LLVM/clang-16 in native_asan job

a440295

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 27, 2023

ci: use LLVM/clang-16 in native_asan job

114fdfe

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request May 1, 2023

ci: use LLVM/clang-16 in native_asan job

68da768

Similar to bitcoin#27298.

fanquake added a commit to fanquake/bitcoin that referenced this pull request May 2, 2023

ci: use LLVM/clang-16 in native_asan job

1232c2f

Similar to bitcoin#27298.

RandyMcMillan pushed a commit to RandyMcMillan/bitcoin that referenced this pull request May 27, 2023

ci: use LLVM/clang-16 in native_fuzz (ASAN) job

f368db2

Similar to bitcoin#27298.

RandyMcMillan pushed a commit to RandyMcMillan/bitcoin that referenced this pull request May 27, 2023

ci: use LLVM/clang-16 in native_asan job

3fd7216

Similar to bitcoin#27298.

maflcko mentioned this pull request Jan 4, 2024

ubsan: misaligned-pointer-use in crc32c/src/crc32c_arm64.cc #29178

Closed

PastaPastaPasta added a commit to dashpay/dash that referenced this pull request Feb 12, 2024

Merge pull request #5867 from UdjinM6/clang16tsan

2f65282

backport: bitcoin#27298 (use clang 16)

bitcoin locked and limited conversation to collaborators Mar 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: Use TSan new runtime (llvm-16, take 3) #27298

ci: Use TSan new runtime (llvm-16, take 3) #27298

Uh oh!

maflcko commented Mar 22, 2023

Uh oh!

DrahtBot commented Mar 22, 2023 •

edited

Loading

Uh oh!

fanquake commented Mar 22, 2023 •

edited

Loading

Uh oh!

dergoegge commented Mar 22, 2023

Uh oh!

dergoegge commented Mar 22, 2023

Uh oh!

maflcko commented Mar 22, 2023

Uh oh!

fanquake commented Mar 22, 2023 •

edited

Loading

Uh oh!

fanquake commented Mar 22, 2023

Uh oh!

maflcko commented Mar 24, 2023

Uh oh!

maflcko commented Mar 24, 2023

Uh oh!

fanquake commented Mar 24, 2023 •

edited

Loading

Uh oh!

fanquake commented Mar 27, 2023

Uh oh!

fanquake commented Mar 27, 2023

Uh oh!

fanquake left a comment

Uh oh!

Uh oh!

ci: Use TSan new runtime (llvm-16, take 3) #27298

ci: Use TSan new runtime (llvm-16, take 3) #27298

Uh oh!

Conversation

maflcko commented Mar 22, 2023

Uh oh!

DrahtBot commented Mar 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews

Conflicts

Uh oh!

fanquake commented Mar 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dergoegge commented Mar 22, 2023

Uh oh!

dergoegge commented Mar 22, 2023

Uh oh!

maflcko commented Mar 22, 2023

Uh oh!

fanquake commented Mar 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fanquake commented Mar 22, 2023

Uh oh!

maflcko commented Mar 24, 2023

Uh oh!

maflcko commented Mar 24, 2023

Uh oh!

fanquake commented Mar 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fanquake commented Mar 27, 2023

Uh oh!

fanquake commented Mar 27, 2023

Uh oh!

fanquake left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DrahtBot commented Mar 22, 2023 •

edited

Loading

fanquake commented Mar 22, 2023 •

edited

Loading

fanquake commented Mar 22, 2023 •

edited

Loading

fanquake commented Mar 24, 2023 •

edited

Loading