tracing: macOS USDT support #25541

kouloumos · 2022-07-04T16:38:28Z

This PR adds macOS support for User-Space, Statically Defined Tracing (USDT) as well as dtrace example scripts based on the existing bpftrace scripts for Linux. This is tested on macOS 10.15.

Overview

The current implementation of USDT only supports Linux as the DTRACE_PROBE*(context, event, ...args) macros are not supported on macOS. As initially referenced in #22238, the process of using USDT probes on macOS is slightly different and it's described in the BUILDING CODE CONTAINING USDT PROBES section in the dtrace(1) manpage.

This more involved process includes

Creating the providers description file util/probes.d which defines the providers and specifies their probes.
Use util/probes.d to generate the header file util/probes.h with the neccesary tracepoints macros which are of the format CONTEXT_EVENT(...args).
Add the tracepoints macros to util/trace.h.

Notes

Part of the macOS process is to create the providers description in a .d file. Therefore, types not supported by the D language (specifically bool and std::byte) cannot be used as part of the probe's signature. On those occasions, in lack of a better solution, only supported types are used and then a patch is applied at the generated util/probes.h file to replace the temporary D-supported types with those that we actually need.

You can reproduce the initial header file by running dtrace -h -s src/util/probes.d -o src/util/probes.h (compiling with that results to errors because of not matching types) and then apply the probes-fix.diff patch with git apply probes-fix.diff.
Edit: util/probes.h is now generated during build time after changes (see #25541 (comment) and #25541 (comment)) that removed the need for the non supported types.

Having the bitcoind binary compiled with USDT support, you can then use the dtrace example scripts connectblock_benchmark.d, log_p2p_traffic.d, log_utxos.d (root privileges needed) to test the macOS support. Extra documentation for those can be found at contrib/tracing as they are functionally the same as the existing bpftrace scripts.

Adding tracepoints to Bitcoin Core (extra steps after this PR)

After this PR when a new tracepoint is added, we will need to

Define the new probe for the tracepoint at util/probes.d together with a new provider if needed.
Add a new #define context_event CONTEXT_EVENT macro at util/trace.h. This might not be final as discussed at tracing: macOS USDT support #25541 (comment).

michaelfolkson · 2022-07-04T16:53:55Z

Concept ACK. It would be great to support tracing on MacOS assuming the maintenance burden is satisfactory.

(The linter checks are failing currently.)

src/util/probes.d

brunoerg · 2022-07-04T17:04:32Z

Concept ACK

configure.ac

0xB10C · 2022-07-05T09:48:58Z

Concept ACK. Thank you for working on this 🚀 I plan to review this. I want to see e.g. what additional things we would need to consider when adding new tracepoints. A few first impressions, notes, and questions:

This looks like a cool PoC, but not really close to being ready for merge, especially with the custom patch :). Could also be a draft PR.
Awesome to see that we can re-use the same TRACEx macros we've already placed in the code. This was how I envisioned macOS (or Windows if possible) tracing support.

and then apply the probes-fix.diff patch with git apply probes-fix.diff.

Would be good find a workaround for this. Did you create the patch manually? Can we change the tracepoints to improve this? I.e. not use bool?
Does src/util/probes.h need to be in version control or can it be generated during build time?
I was able to generate src/util/probes.h from src/util/probes.d on Linux, with dtrace supplied by Systemtap. However, it looks very different from your probes.h file. See this gits.
It would be ideal if we could have a probes.d file that works for both Linux and macOS.
The *_ENABLED() in probes.h reminded me of https://eklitzke.org/how-sytemtap-userspace-probes-work
We should check the macOS GUIX builds at some point to see if tracing works with them.
We'd probably want to document these steps in doc/tracing.md once we've figured something out that works well and doesn't add to much maintenance. Also, the *.d scripts can be mentioned under the example section.

cc @jb55

src/util/probes.d

0xB10C · 2022-07-05T09:53:43Z

src/util/trace.h

+// since the dtrace macros are automatically generated in uppercase, additional
+// macros are needed to translate the lowercase context & event names into the
+// required uppercase CONTEXT_EVENT macros


we could change that, if this makes it a lot easier for macOS. This would probably be an API break, but I think the tracepoints are still experimental enough to be able to do it.

Changing to uppercase was part of my initial implementation but I changed my mind.
Even if we need those uppercase macros for macOS, the actual tracepoints for macOS would still be of the format net:inbound_message. But my understanding is that if we change TRACEx to TRACE6(NET, INBOUND_MESSSAGE, ..args) the Linux tracepoints would be NET:INBOUND_MESSSAGE thus not having a unified format across systems.

kouloumos · 2022-07-05T14:57:28Z

Thank you for the review @0xB10C! I am not happy with that custom patch either, but this being my first interaction with the build system, I was not sure what is consider acceptable. In retrospect, it is not and it adds unnecessary maintenance burden.

Would be good find a workaround for this. Did you create the patch manually? Can we change the tracepoints to improve this? I.e. not use bool?

If what you mentioned at #25541 (comment) is acceptable and we find a way to not use bool for the tracepoints, the generation of the header file could possibly become part of the build process. I avoided thinking about changing the bool type, mostly because of the overhead that this might introduced.

Does src/util/probes.h need to be in version control or can it be generated during build time?
...

The *_ENABLED() in probes.h reminded me https://eklitzke.org/how-sytemtap-userspace-probes-work

Looking into the probes_sdt branch in that article, it seems that generating it during build time might be possible. This would possibly result to a probes.d file that works for both Linux and macOS. I initially avoided going that route because of the currently required changes (patch) after the file generation.

I was able to generate src/util/probes.h from src/util/probes.d on Linux, with dtrace supplied by Systemtap. However, it looks very different from your probes.h file. See this gits.

I think being different makes sense as the resulting file has a different target os? My probes.h file is not the actual result of the command, as the committed file is after the manual changes shown in the patch. I've updated the patch link with the initial (before applying the patch) probes.h file.

We'd probably want to document these steps in doc/tracing.md once we've figured something out that works well and doesn't add to much maintenance. Also, the *.d scripts can be mentioned under the example section.

Yes, I plan to enhance the docs as soon as this works well.

I'll convert this to a draft PR until I find a way to get rid of the patch and in the meantime fix the failing checks.
Best case scenario: replace bool > generate util/probes during build time > 🚀

src/util/probes.d

0xB10C · 2022-07-06T19:43:03Z

src/net_processing.cpp

@@ -4275,7 +4275,7 @@ bool PeerManagerImpl::ProcessMessages(CNode* pfrom, std::atomic<bool>& interrupt
        pfrom->ConnectionTypeAsString().c_str(),
        msg.m_type.c_str(),
        msg.m_recv.size(),
-        msg.m_recv.data()
+        (unsigned char*)(msg.m_recv.data())


Note to self: I want to check if this causes a difference in the number of instructions for this tracepoint. I think a reinterpret_cast could also work here as an alternative:

Unlike static_cast, but like const_cast, the reinterpret_cast expression does not compile to any CPU instructions [..]

DrahtBot · 2022-07-07T03:33:13Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	michaelfolkson, brunoerg, 0xB10C, RandyMcMillan

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#26593 (tracing: Only prepare tracepoint arguments when actually tracing by 0xB10C)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

This deduplicates the TRACEx macros by using systemtaps [STAP_PROBEV] variadic macro instead of the [DTrace compability DTRACE_PROBE] macros. Bitcoin Core never had DTrace tracepoints, so we don't need to use the drop-in replacement for it. As noted in bitcoin#25541[1], these macros aren't compatibile with DTrace on macOS anyway. The (currently unused) TRACE() macro is renamed to TRACEPOINT0(). The deduplication is helpful for the next commit too. This also renames the TRACEx macro to TRACEPOINT to clarify what the macro does: inserting a tracepoint vs tracing (logging) something. [STAP_PROBEV]: https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h;h=24d5e01c37805e55c36f7202e5d4e821b85167a1;hb=ecab2afea46099b4e7dfd551462689224afdbe3a#l407 [DTrace compability DTRACE_PROBE]: https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h;h=24d5e01c37805e55c36f7202e5d4e821b85167a1;hb=ecab2afea46099b4e7dfd551462689224afdbe3a#l490 [1]: https://github.com/bitcoin/bitcoin/pull/25541/files#diff-553886c5f808e01e3452c7b21e879cc355da388ef7680bf310f6acb926d43266R30-R31

Before this commit, we would always prepare tracepoint arguments regardless of the tracepoint being used or not. While we already made sure not to include expensive arguments in our tracepoints, this commit introduces gating to make sure the arguments are only prepared if the tracepoints are actually used. This is a win-win improvement to our tracing framework. For users not interested in tracing, the overhead is reduced to a cheap 'greater than 0' compare. As the semaphore-gating technique used here is available in bpftrace, bcc, and libbpf, users interested in tracing don't have to change their tracing scripts while profiting from potential future tracepoints passing slightly more expensive arguments. In the context of bitcoin#26531's mempool tracepoints, one example could be passing serialized transactions for RBF replacements. Under the hood, the semaphore-gating works by placing a 2-byte semaphore in the '.probes' ELF section. The address of the semaphore is contained in the ELF note providing the tracepoint information (`readelf -n ./src/bitcoind | grep NT_STAPSDT`). Tracing toolkits like bpftrace, bcc, and libbpf increase the semaphore at the address upon attaching to the tracepoint. We only prepare the arguments and reach the tracepoint if the semaphore is greater than zero. The semaphore is decreased when detaching from the tracepoint. The implementation of the TRACEPOINT_ACTIVE macro is flexible enough that it can be adapted for macOS dtrace tracing (bitcoin#25541). With dtrace, [CONTEXT]_[EVENT]_ENABLED() can be used to gate the tracepoints. While the TRACEPOINT0(context, event) macro without arguments doesn't need a check if it's active or not (no arguments can be prepared and passed), it does require a TRACEPOINT_SEMAPHORE() due to _SDT_HAS_SEMAPHORES requiring all tracepoints to have a semaphore. As the check is cheap, a check is added to not cause problems with the semaphore being unused. This also extends the "Adding a new tracepoint" documentation to include information about the semaphores and updated step-by-step instructions on how to add a new tracepoint.

This deduplicates the TRACEx macros by using systemtaps [STAP_PROBEV] variadic macro instead of the [DTrace compability DTRACE_PROBE] macros. Bitcoin Core never had DTrace tracepoints, so we don't need to use the drop-in replacement for it. As noted in bitcoin#25541[1], these macros aren't compatibile with DTrace on macOS anyway. The (currently unused) TRACE() macro is renamed to TRACEPOINT0(). The deduplication is helpful for the next commit too. This also renames the TRACEx macro to TRACEPOINT to clarify what the macro does: inserting a tracepoint vs tracing (logging) something. [STAP_PROBEV]: https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h;h=24d5e01c37805e55c36f7202e5d4e821b85167a1;hb=ecab2afea46099b4e7dfd551462689224afdbe3a#l407 [DTrace compability DTRACE_PROBE]: https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h;h=24d5e01c37805e55c36f7202e5d4e821b85167a1;hb=ecab2afea46099b4e7dfd551462689224afdbe3a#l490 [1]: https://github.com/bitcoin/bitcoin/pull/25541/files#diff-553886c5f808e01e3452c7b21e879cc355da388ef7680bf310f6acb926d43266R30-R31

Before this commit, we would always prepare tracepoint arguments regardless of the tracepoint being used or not. While we already made sure not to include expensive arguments in our tracepoints, this commit introduces gating to make sure the arguments are only prepared if the tracepoints are actually used. This is a win-win improvement to our tracing framework. For users not interested in tracing, the overhead is reduced to a cheap 'greater than 0' compare. As the semaphore-gating technique used here is available in bpftrace, bcc, and libbpf, users interested in tracing don't have to change their tracing scripts while profiting from potential future tracepoints passing slightly more expensive arguments. In the context of bitcoin#26531's mempool tracepoints, one example could be passing serialized transactions for RBF replacements. Under the hood, the semaphore-gating works by placing a 2-byte semaphore in the '.probes' ELF section. The address of the semaphore is contained in the ELF note providing the tracepoint information (`readelf -n ./src/bitcoind | grep NT_STAPSDT`). Tracing toolkits like bpftrace, bcc, and libbpf increase the semaphore at the address upon attaching to the tracepoint. We only prepare the arguments and reach the tracepoint if the semaphore is greater than zero. The semaphore is decreased when detaching from the tracepoint. The implementation of the TRACEPOINT_ACTIVE macro is flexible enough that it can be adapted for macOS dtrace tracing (bitcoin#25541). With dtrace, [CONTEXT]_[EVENT]_ENABLED() can be used to gate the tracepoints. While the TRACEPOINT0(context, event) macro without arguments doesn't need a check if it's active or not (no arguments can be prepared and passed), it does require a TRACEPOINT_SEMAPHORE() due to _SDT_HAS_SEMAPHORES requiring all tracepoints to have a semaphore. As the check is cheap, a check is added to not cause problems with the semaphore being unused. This also extends the "Adding a new tracepoint" documentation to include information about the semaphores and updated step-by-step instructions on how to add a new tracepoint.

This deduplicates the TRACEx macros by using systemtaps [STAP_PROBEV] variadic macro instead of the [DTrace compability DTRACE_PROBE] macros. Bitcoin Core never had DTrace tracepoints, so we don't need to use the drop-in replacement for it. As noted in bitcoin#25541[1], these macros aren't compatibile with DTrace on macOS anyway. The (currently unused) TRACE() macro is renamed to TRACEPOINT0(). The deduplication is helpful for the next commit too. This also renames the TRACEx macro to TRACEPOINT to clarify what the macro does: inserting a tracepoint vs tracing (logging) something. [STAP_PROBEV]: https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h;h=24d5e01c37805e55c36f7202e5d4e821b85167a1;hb=ecab2afea46099b4e7dfd551462689224afdbe3a#l407 [DTrace compability DTRACE_PROBE]: https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h;h=24d5e01c37805e55c36f7202e5d4e821b85167a1;hb=ecab2afea46099b4e7dfd551462689224afdbe3a#l490 [1]: https://github.com/bitcoin/bitcoin/pull/25541/files#diff-553886c5f808e01e3452c7b21e879cc355da388ef7680bf310f6acb926d43266R30-R31

Before this commit, we would always prepare tracepoint arguments regardless of the tracepoint being used or not. While we already made sure not to include expensive arguments in our tracepoints, this commit introduces gating to make sure the arguments are only prepared if the tracepoints are actually used. This is a win-win improvement to our tracing framework. For users not interested in tracing, the overhead is reduced to a cheap 'greater than 0' compare. As the semaphore-gating technique used here is available in bpftrace, bcc, and libbpf, users interested in tracing don't have to change their tracing scripts while profiting from potential future tracepoints passing slightly more expensive arguments. In the context of bitcoin#26531's mempool tracepoints, one example could be passing serialized transactions for RBF replacements. Under the hood, the semaphore-gating works by placing a 2-byte semaphore in the '.probes' ELF section. The address of the semaphore is contained in the ELF note providing the tracepoint information (`readelf -n ./src/bitcoind | grep NT_STAPSDT`). Tracing toolkits like bpftrace, bcc, and libbpf increase the semaphore at the address upon attaching to the tracepoint. We only prepare the arguments and reach the tracepoint if the semaphore is greater than zero. The semaphore is decreased when detaching from the tracepoint. The implementation of the TRACEPOINT_ACTIVE macro is flexible enough that it can be adapted for macOS dtrace tracing (bitcoin#25541). With dtrace, [CONTEXT]_[EVENT]_ENABLED() can be used to gate the tracepoints. While the TRACEPOINT0(context, event) macro without arguments doesn't need a check if it's active or not (no arguments can be prepared and passed), it does require a TRACEPOINT_SEMAPHORE() due to _SDT_HAS_SEMAPHORES requiring all tracepoints to have a semaphore. As the check is cheap, a check is added to not cause problems with the semaphore being unused. This also extends the "Adding a new tracepoint" documentation to include information about the semaphores and updated step-by-step instructions on how to add a new tracepoint.

This deduplicates the TRACEx macros by using systemtaps [STAP_PROBEV] variadic macro instead of the [DTrace compability DTRACE_PROBE] macros. Bitcoin Core never had DTrace tracepoints, so we don't need to use the drop-in replacement for it. As noted in bitcoin#25541[1], these macros aren't compatibile with DTrace on macOS anyway. The (currently unused) TRACE() macro is renamed to TRACEPOINT0(). The deduplication is helpful for the next commit too. This also renames the TRACEx macro to TRACEPOINT to clarify what the macro does: inserting a tracepoint vs tracing (logging) something. [STAP_PROBEV]: https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h;h=24d5e01c37805e55c36f7202e5d4e821b85167a1;hb=ecab2afea46099b4e7dfd551462689224afdbe3a#l407 [DTrace compability DTRACE_PROBE]: https://sourceware.org/git/?p=systemtap.git;a=blob;f=includes/sys/sdt.h;h=24d5e01c37805e55c36f7202e5d4e821b85167a1;hb=ecab2afea46099b4e7dfd551462689224afdbe3a#l490 [1]: https://github.com/bitcoin/bitcoin/pull/25541/files#diff-553886c5f808e01e3452c7b21e879cc355da388ef7680bf310f6acb926d43266R30-R31

Before this commit, we would always prepare tracepoint arguments regardless of the tracepoint being used or not. While we already made sure not to include expensive arguments in our tracepoints, this commit introduces gating to make sure the arguments are only prepared if the tracepoints are actually used. This is a win-win improvement to our tracing framework. For users not interested in tracing, the overhead is reduced to a cheap 'greater than 0' compare. As the semaphore-gating technique used here is available in bpftrace, bcc, and libbpf, users interested in tracing don't have to change their tracing scripts while profiting from potential future tracepoints passing slightly more expensive arguments. In the context of bitcoin#26531's mempool tracepoints, one example could be passing serialized transactions for RBF replacements. Under the hood, the semaphore-gating works by placing a 2-byte semaphore in the '.probes' ELF section. The address of the semaphore is contained in the ELF note providing the tracepoint information (`readelf -n ./src/bitcoind | grep NT_STAPSDT`). Tracing toolkits like bpftrace, bcc, and libbpf increase the semaphore at the address upon attaching to the tracepoint. We only prepare the arguments and reach the tracepoint if the semaphore is greater than zero. The semaphore is decreased when detaching from the tracepoint. The implementation of the TRACEPOINT_ACTIVE macro is flexible enough that it can be adapted for macOS dtrace tracing (bitcoin#25541). With dtrace, [CONTEXT]_[EVENT]_ENABLED() can be used to gate the tracepoints. While the TRACEPOINT0(context, event) macro without arguments doesn't need a check if it's active or not (no arguments can be prepared and passed), it does require a TRACEPOINT_SEMAPHORE() due to _SDT_HAS_SEMAPHORES requiring all tracepoints to have a semaphore. As the check is cheap, a check is added to not cause problems with the semaphore being unused. This also extends the "Adding a new tracepoint" documentation to include information about the semaphores and updated step-by-step instructions on how to add a new tracepoint.

achow101 · 2023-04-25T15:15:19Z

What's the status of this?

kouloumos · 2023-05-02T12:30:55Z

What's the status of this?

It seems that there isn't enough interest for this change. But anyway. with the current status of #26593, it doesn't make sense for this PR to be dealt with before that one is merged. Additionally, there is a chance that #26593 will make it easier to reason for macOS USDT support if what I proposed gets into the final PR.
Therefore, marking it as draft for now.

fanquake · 2024-03-06T14:05:04Z

Ok, going to close this for now, until these is sort of outcome out of #26593. Can be reopened if/when required.

fanquake added the macOS label Jul 4, 2022

brunoerg reviewed Jul 4, 2022

View reviewed changes

src/util/probes.d Outdated Show resolved Hide resolved

hebasto reviewed Jul 4, 2022

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

kouloumos force-pushed the macos-usdt-support branch 2 times, most recently from 05f7baf to 1aacde1 Compare July 4, 2022 19:28

0xB10C reviewed Jul 5, 2022

View reviewed changes

src/util/probes.d Outdated Show resolved Hide resolved

0xB10C reviewed Jul 5, 2022

View reviewed changes

bitcoin deleted a comment from GrowingDeere Jul 5, 2022

laanwj added the Scripts and tools label Jul 5, 2022

kouloumos marked this pull request as draft July 5, 2022 15:10

kouloumos force-pushed the macos-usdt-support branch 3 times, most recently from 8edd380 to 64d0862 Compare July 6, 2022 19:03

0xB10C reviewed Jul 6, 2022

View reviewed changes

src/util/probes.d Show resolved Hide resolved

0xB10C reviewed Jul 6, 2022

View reviewed changes

DrahtBot mentioned this pull request Jul 7, 2022

build: Treat obj/build.h as an intermediate file #22624

Closed

kouloumos force-pushed the macos-usdt-support branch 2 times, most recently from f3734d8 to c6ce901 Compare July 7, 2022 13:28

tracing: macOS USDT support

6e6012a

kouloumos force-pushed the macos-usdt-support branch 6 times, most recently from 4a23882 to 8498bcf Compare July 7, 2022 15:42

0xB10C mentioned this pull request Apr 13, 2023

build: Detect USDT the same way how it is used in the code #27458

Merged

kouloumos marked this pull request as draft May 2, 2023 12:31

DrahtBot added CI failed and removed CI failed labels May 30, 2023

DrahtBot added CI failed and removed CI failed labels Jul 5, 2023

DrahtBot mentioned this pull request Aug 6, 2023

util/blockstorage: Add TRACE_RAII, slightly faster -reindex-chainstate with CBufferedFile #28226

Closed

DrahtBot added CI failed and removed CI failed labels Oct 25, 2023

DrahtBot added the CI failed label Nov 27, 2023

DrahtBot removed the CI failed label Dec 10, 2023

DrahtBot added the CI failed label Jan 1, 2024

DrahtBot added CI failed and removed CI failed labels Jan 9, 2024

fanquake closed this Mar 6, 2024

0xB10C mentioned this pull request Nov 11, 2024

Tracepoint Interface Tracking Issue #31274

Open

yuvicc mentioned this pull request Jan 27, 2025

doc: update translation generation cmake example #31731

Merged

bitcoin locked and limited conversation to collaborators Mar 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tracing: macOS USDT support #25541

tracing: macOS USDT support #25541

Uh oh!

kouloumos commented Jul 4, 2022 •

edited

Loading

Uh oh!

michaelfolkson commented Jul 4, 2022

Uh oh!

Uh oh!

brunoerg commented Jul 4, 2022

Uh oh!

Uh oh!

0xB10C commented Jul 5, 2022 •

edited

Loading

Uh oh!

Uh oh!

0xB10C Jul 5, 2022

Uh oh!

kouloumos Jul 5, 2022

Uh oh!

kouloumos commented Jul 5, 2022

Uh oh!

Uh oh!

0xB10C Jul 6, 2022

Uh oh!

DrahtBot commented Jul 7, 2022 •

edited

Loading

Uh oh!

achow101 commented Apr 25, 2023

Uh oh!

kouloumos commented May 2, 2023

Uh oh!

fanquake commented Mar 6, 2024

Uh oh!

Uh oh!

tracing: macOS USDT support #25541

tracing: macOS USDT support #25541

Uh oh!

Conversation

kouloumos commented Jul 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Notes

Adding tracepoints to Bitcoin Core (extra steps after this PR)

Uh oh!

michaelfolkson commented Jul 4, 2022

Uh oh!

Uh oh!

brunoerg commented Jul 4, 2022

Uh oh!

Uh oh!

0xB10C commented Jul 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

0xB10C Jul 5, 2022

Choose a reason for hiding this comment

Uh oh!

kouloumos Jul 5, 2022

Choose a reason for hiding this comment

Uh oh!

kouloumos commented Jul 5, 2022

Uh oh!

Uh oh!

0xB10C Jul 6, 2022

Choose a reason for hiding this comment

Uh oh!

DrahtBot commented Jul 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews

Conflicts

Uh oh!

achow101 commented Apr 25, 2023

Uh oh!

kouloumos commented May 2, 2023

Uh oh!

fanquake commented Mar 6, 2024

Uh oh!

Uh oh!

kouloumos commented Jul 4, 2022 •

edited

Loading

0xB10C commented Jul 5, 2022 •

edited

Loading

DrahtBot commented Jul 7, 2022 •

edited

Loading