Skip to content

Tokio runtime periodically freezes in Linux kernel 6.14.4+  #7335

@macladson

Description

@macladson

Version
1.45.0 (I also confirmed 1.1.1 is also affected)

Platform
Linux thinkpad 6.14.6-arch1-2 #1 SMP PREEMPT_DYNAMIC Tue, 29 Apr 2025 09:23:13 +0000 x86_64 GNU/Linux

So far I have confirmed that this affects:

  • Arch Linux running 6.14.4 -> 6.14.6
  • Fedora 41 running 6.14.6
  • Debian experimental (running 6.14.6)

This does not affect Arch Linux running the LTS kernel (currently 6.12.28).

It affects both AMD and Intel CPUs although systems with more resources seem to take longer to freeze.
However, I have been unable to reproduce this on systems with only 2 (or less) available threads so it's possible it requires higher thread counts.

Description
The tokio runtime freezes under high lock contention.
It freezes with low task counts (10) and very high task counts (100,000) but 1000 tasks seems to be the sweetspot where it reproduces the issue most quickly.

I have found that on my 4 core Intel Thinkpad running Arch Linux with the 6.14.4 kernel, I am able to consistently reproduce it within a few minutes. Other systems (especially those with more resources) can take much longer (sometimes 12+ hours).

See this repo for a very simple reproduction: https://github.com/macladson/tokio-lock-example

It's not a traditional deadlock, because the runtime can be awoken by certain signals. For example it will wake up when:

  • Attaching then detaching gdb (only attaching doesn't wake it)
  • Receiving an API call in a traditional application
  • Your machine entering sleep mode/suspend then waking up

One notable thing is that if I remove the call to tokio::time::sleep the freeze never occurs.

I'm not an expert using tokio by any means but I believe the code I presented should ordinarily work.

Given that this only seems to affect the newer kernels, this may be a kernel regression rather than a tokio-specific bug, but either way it felt serious enough to raise here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-tokioArea: The main tokio crateC-bugCategory: This is a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions