Liveness probes for the RabbitMQ pod are failing occasionally in production, sometimes leading to the pod being terminated and replaced. Other than interruptions caused by the roll-over, there are no signs of problems with the service. Notably, the celery worker is processing jobs without apparent interruption, which indicates that the message queue is operating. RabbitMQ itself does not report any errors, and its memory / CPU usage are not remarkable. There are some indications that the k8s node might be busy at the time of the mq pod restart (synthetics checks had slow responses at around the same time).

My suspicion is that once in a while, perhaps during heavy load, the rabbitmq-diagnostics ping command we use is taking too long to execute. We're using a short (5s) timeout on the liveness probe. This bumps the timeout to 30 seconds on the ping command, which was using its default infinite timeout. The k8s livenessProbe config timeout is set to 35 seconds to allow time for the command to start / exit.