What version of Go are you using (go version)?

go version go1.20.1 linux/amd64

Does this issue reproduce with the latest release?

Yes, with 1.20.1

What operating system and processor architecture are you using (go env)?

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

What did you do?

After upgrading avalanchego to use go1.20.1 instead of 1.19.6, the binary now randomly halts (all goroutines stop and the program becomes unresponsive) after about 20-30 minutes of runtime on go1.20.1 linux/amd64.

Notably, the binary does not panic or exit at this point (as you would expect if it was deadlocked). It just stays pegged to ~30% CPU and spins indefinitely on something (but all async processes halt):

The output of grs -r on dlv is attached from the point of stall:
goroutines.txt

What did you expect to see?

The program to work like it did on v1.19.6 (not halt without warning or panic). To run this test (and hopefully avoid wasting your time), we compiled the exact same code in both 1.19.6 and 1.20.1 (this only occurred in 1.20.1).

What did you see instead?

The program halted without emitting a panic or any other info that could be used to diagnose the issue. All metrics-related goroutines that would normally inform of system level processes also stopped running (I.e. global program halt).