-
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
Description
Version
1.45.0
(I also confirmed 1.1.1
is also affected)
Platform
Linux thinkpad 6.14.6-arch1-2 #1 SMP PREEMPT_DYNAMIC Tue, 29 Apr 2025 09:23:13 +0000 x86_64 GNU/Linux
So far I have confirmed that this affects:
- Arch Linux running
6.14.4
->6.14.6
- Fedora 41 running
6.14.6
- Debian experimental (running
6.14.6
)
This does not affect Arch Linux running the LTS kernel (currently 6.12.28
).
It affects both AMD and Intel CPUs although systems with more resources seem to take longer to freeze.
However, I have been unable to reproduce this on systems with only 2 (or less) available threads so it's possible it requires higher thread counts.
Description
The tokio
runtime freezes under high lock contention.
It freezes with low task counts (10) and very high task counts (100,000) but 1000 tasks seems to be the sweetspot where it reproduces the issue most quickly.
I have found that on my 4 core Intel Thinkpad running Arch Linux with the 6.14.4
kernel, I am able to consistently reproduce it within a few minutes. Other systems (especially those with more resources) can take much longer (sometimes 12+ hours).
See this repo for a very simple reproduction: https://github.com/macladson/tokio-lock-example
It's not a traditional deadlock, because the runtime can be awoken by certain signals. For example it will wake up when:
- Attaching then detaching
gdb
(only attaching doesn't wake it) - Receiving an API call in a traditional application
- Your machine entering sleep mode/suspend then waking up
One notable thing is that if I remove the call to tokio::time::sleep
the freeze never occurs.
I'm not an expert using tokio
by any means but I believe the code I presented should ordinarily work.
Given that this only seems to affect the newer kernels, this may be a kernel regression rather than a tokio
-specific bug, but either way it felt serious enough to raise here.