-
Notifications
You must be signed in to change notification settings - Fork 897
Description
Summary
The Validator Client on Linux with kernel versions 6.14.4
-> 6.14.7
will intermittently freeze, stopping it from performing its duties.
If you're running a distro which closely follows Linux mainline (such as Arch Linux or Fedora) you may be affected.
Run uname -r
to check if your kernel version is in the affected range.
Note: Ubuntu 24.04 and older use older kernels and thus are not affected. However, Ubuntu 25.04 is currently running the 6.14.0 kernel which is unaffected but it is possible that a future package upgrade will include one of the affected kernels so upgrade with caution.
Solutions and Workarounds
This bug, caused by changes to the eventpoll
code, has already been patched in the Linux mainline kernel and will be fixed in 6.14.8+
.
Once your distro allows you to update the kernel to 6.14.8
you can safely do so.
In the meantime, if you are running an affected kernel version you have a few options:
Install an LTS kernel
The procedure for this will differ depending on your distro but the below example is the instructions for Arch Linux:
sudo pacman -S linux-lts
If you are using systemd-boot
, it should automatically generate the corresponding bootloader entries.
If you are using grub
you will need to regenerate them:
sudo grub-mkconfig -o /boot/grub/grub.cfg
Reboot and you should see linux-lts
included in the grub
menu.
Downgrade your kernel
This will vary depending on your distro and for some distros it is very involved. Here are the instructions for Fedora 41:
# Find available kernels
sudo dnf list kernel --showduplicates
# Install a specific kernel. For example:
sudo dnf install kernel-6.11.4-301.fc41
grub
entries will automatically be added, so reboot and select the new kernel from the list.
Run an API polling script
If you do not want to touch your kernel in case you break something there is a simple bash
script you can run instead.
Due to the internals of eventpoll
, when the VC receives an API call, it will wake from its freeze.
Because of this we can use a script running in the background which continuously polls the VC. Here is an example of such a script:
while sleep 5; do curl -s --fail "http://localhost:5062/lighthouse/auth" > /dev/null && echo "polled at $(date)"; done
This will keep the VC awake.
Note that running a full VC metrics server with Grafana polling the VC will also keep it awake for the same reason.
Acknowledgments
A huge thank you to the users on Discord who discovered this issue and assisted in diagnosis and testing, particularly @smooth.ninja and @ChosunOne.
See tokio-rs/tokio#7335 and #7403 for more details