Various error logs in agent and operator when running GatewayAPI #35595
Description
Noticed in CI on main after collecting sysdumps: https://github.com/cilium/cilium/actions/runs/11553111281.
Example sysdump: cilium-sysdump-out-standard-false.zip
We can see a bunch of errors in the agent logs:
$ git grep level=err logs-cilium-zt8gf-cilium-agent-20241028-114847.log
logs-cilium-zt8gf-cilium-agent-20241028-114847.log:2024-10-28T11:47:57.752720231Z time=2024-10-28T11:47:57Z level=error msg="Observer job errored" module=agent.controlplane.ciliumenvoyconfig name=cec-resource-events func=ciliumenvoyconfig.(*ciliumEnvoyConfigReconciler).handleCECEvent error="failed to handle CEC upsert: failed to redirect k8s services to Envoy in CEC Update: could not retrieve service details for service gateway-conformance-infra/cilium-gateway-httproute-listener-hostname-matching"
logs-cilium-zt8gf-cilium-agent-20241028-114847.log:2024-10-28T11:48:22.191260248Z time="2024-10-28T11:48:22.191217206Z" level=error msg="failed to sync headless service" error="retrieved nil details for Service gateway-conformance-infra/headless" key=gateway-conformance-infra/cilium-gateway-same-namespace subsys=ciliumenvoyconfig svcKey=gateway-conformance-infra/headless-smnd9
logs-cilium-zt8gf-cilium-agent-20241028-114847.log:2024-10-28T11:48:22.191634028Z time=2024-10-28T11:48:22Z level=error msg="Observer job errored" module=agent.controlplane.ciliumenvoyconfig name=headless-endpoint-events func=ciliumenvoyconfig.(*ciliumEnvoyConfigReconciler).syncEndpoints error="failed to sync headless service (gateway-conformance-infra/cilium-gateway-same-namespace): retrieved nil details for Service gateway-conformance-infra/headless"
logs-cilium-zt8gf-cilium-agent-20241028-114847.log:2024-10-28T11:48:22.206377758Z time="2024-10-28T11:48:22.206162874Z" level=error msg="failed to sync headless service" error="retrieved nil details for Service gateway-conformance-infra/manual-endpointslices" key=gateway-conformance-infra/cilium-gateway-same-namespace subsys=ciliumenvoyconfig svcKey=gateway-conformance-infra/manual-endpointslices-ip4
logs-cilium-zt8gf-cilium-agent-20241028-114847.log:2024-10-28T11:48:22.206810758Z time=2024-10-28T11:48:22Z level=error msg="Observer job errored" module=agent.controlplane.ciliumenvoyconfig name=headless-endpoint-events func=ciliumenvoyconfig.(*ciliumEnvoyConfigReconciler).syncEndpoints error="failed to sync headless service (gateway-conformance-infra/cilium-gateway-same-namespace): retrieved nil details for Service gateway-conformance-infra/manual-endpointslices"
logs-cilium-zt8gf-cilium-agent-20241028-114847.log:2024-10-28T11:48:22.256220848Z time=2024-10-28T11:48:22Z level=error msg="Observer job errored" module=agent.controlplane.ciliumenvoyconfig name=cec-resource-events func=ciliumenvoyconfig.(*ciliumEnvoyConfigReconciler).handleCECEvent error="failed to handle CEC upsert: failed to redirect k8s services to Envoy in CEC Update: retrieved nil details for Service gateway-conformance-infra/manual-endpointslices"
logs-cilium-zt8gf-cilium-agent-20241028-114847.log:2024-10-28T11:48:22.274124000Z time=2024-10-28T11:48:22Z level=error msg="Observer job errored" module=agent.controlplane.ciliumenvoyconfig name=cec-resource-events func=ciliumenvoyconfig.(*ciliumEnvoyConfigReconciler).handleCECEvent error="failed to handle CEC upsert: failed to redirect k8s services to Envoy in CEC Update: retrieved nil details for Service gateway-conformance-infra/manual-endpointslices"
And a bunch more in the operator logs (some are the same):
$ git grep level=err logs-cilium-operator-b969b9997-sb4rt-cilium-operator-20241028-114847.log
logs-cilium-operator-b969b9997-sb4rt-cilium-operator-20241028-114847.log:2024-10-28T11:47:25.996482739Z time=2024-10-28T11:47:25Z level=error msg="Unable to ensure CiliumEnvoyConfig" module=operator.operator-controlplane.leader-lifecycle.gateway-api controller=gateway resource=gateway-conformance-infra/gateway-remove-listener error="ciliumenvoyconfigs.cilium.io \"cilium-gateway-gateway-remove-listener\" not found"
logs-cilium-operator-b969b9997-sb4rt-cilium-operator-20241028-114847.log:2024-10-28T11:47:31.449905700Z time=2024-10-28T11:47:31Z level=error msg="The gateway static address is not yet supported" module=operator.operator-controlplane.leader-lifecycle.gateway-api controller=gateway resource=gateway-conformance-infra/gateway-static-addresses error="address type is not supported"
logs-cilium-operator-b969b9997-sb4rt-cilium-operator-20241028-114847.log:2024-10-28T11:47:31.460762527Z time=2024-10-28T11:47:31Z level=error msg="Reconciler error" module=operator.operator-controlplane.leader-lifecycle.controller-runtime controller=gateway controllerGroup=gateway.networking.k8s.io controllerKind=Gateway Gateway.name=gateway-static-addresses Gateway.namespace=gateway-conformance-infra namespace=gateway-conformance-infra name=gateway-static-addresses reconcileID=cbac8f01-e24e-427a-9a25-da9ff228334a error="address type is not supported"
logs-cilium-operator-b969b9997-sb4rt-cilium-operator-20241028-114847.log:2024-10-28T11:47:31.461096011Z time=2024-10-28T11:47:31Z level=error msg="The gateway static address is not yet supported" module=operator.operator-controlplane.leader-lifecycle.gateway-api controller=gateway resource=gateway-conformance-infra/gateway-static-addresses error="address type is not supported"
cc @sayboras