Warnings about Hubble Relay connectivity issues #36070
Description
The following warnings happen fairly often in the Clustermesh end-to-end workflow:
level=warning msg="Error while receiving peer change notification; will try again after the timeout has expired" connection timeout=5s error="rpc error: code = Unavailable desc = error reading from server: EOF" subsys=hubble-relay
[...]
level=warning msg="Failed to create gRPC client" address="172.18.0.5:4244" error="connection error: desc = \"transport: error while dialing: dial tcp 172.18.0.5:4244: connect: connection refused\"" hubble-tls=true next-try-in=1s peer=cluster2-with-long-name-01234567/cluster2-11807468374-worker2 subsys=hubble-relay
[...]
level=warning msg="Failed to create peer client for peers synchronization; will try again after the timeout has expired" error="connection error: desc = \"transport: error while dialing: dial tcp 10.245.87.91:443: connect: connection refused\"" subsys=hubble-relay target="hubble-peer.kube-system.svc.cluster.local:443"
Examples: https://github.com/cilium/cilium/actions/runs/11807468374 and https://github.com/cilium/cilium/actions/runs/11800231233/job/32871039983.