Skip to content

Validator Client intermittently freezes on Linux kernel 6.14.4 -> 6.14.7 #7403

@michaelsproul

Description

@michaelsproul

Summary

The Validator Client on Linux with kernel versions 6.14.4 -> 6.14.7 will intermittently freeze, stopping it from performing its duties.

If you're running a distro which closely follows Linux mainline (such as Arch Linux or Fedora) you may be affected.

Run uname -r to check if your kernel version is in the affected range.

Note: Ubuntu 24.04 and older use older kernels and thus are not affected. However, Ubuntu 25.04 is currently running the 6.14.0 kernel which is unaffected but it is possible that a future package upgrade will include one of the affected kernels so upgrade with caution.

Solutions and Workarounds

This bug, caused by changes to the eventpoll code, has already been patched in the Linux mainline kernel and will be fixed in 6.14.8+.

Once your distro allows you to update the kernel to 6.14.8 you can safely do so.

In the meantime, if you are running an affected kernel version you have a few options:

Install an LTS kernel

The procedure for this will differ depending on your distro but the below example is the instructions for Arch Linux:

sudo pacman -S linux-lts

If you are using systemd-boot, it should automatically generate the corresponding bootloader entries.

If you are using grub you will need to regenerate them:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Reboot and you should see linux-lts included in the grub menu.

Downgrade your kernel

This will vary depending on your distro and for some distros it is very involved. Here are the instructions for Fedora 41:

# Find available kernels
sudo dnf list kernel --showduplicates

# Install a specific kernel. For example:
sudo dnf install kernel-6.11.4-301.fc41

grub entries will automatically be added, so reboot and select the new kernel from the list.

Run an API polling script

If you do not want to touch your kernel in case you break something there is a simple bash script you can run instead.

Due to the internals of eventpoll, when the VC receives an API call, it will wake from its freeze.

Because of this we can use a script running in the background which continuously polls the VC. Here is an example of such a script:

while sleep 5; do curl -s --fail "http://localhost:5062/lighthouse/auth" > /dev/null && echo "polled at $(date)"; done

This will keep the VC awake.
Note that running a full VC metrics server with Grafana polling the VC will also keep it awake for the same reason.

Acknowledgments

A huge thank you to the users on Discord who discovered this issue and assisted in diagnosis and testing, particularly @smooth.ninja and @ChosunOne.


See tokio-rs/tokio#7335 and #7403 for more details

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingval-clientRelates to the validator client binary

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions