The blamed commit introduced the removal of a service entry from the
internal service cache data structure upon reception of a deletion
event for remote endpoints from a remote cluster, if the service is
no longer associated with either a local endpointslice or remote
endpoints from any other cluster.

However, this is not correct, and can lead to an unrecoverable situation
(until a service update is performed, or Cilium agents are restarted),
because subsequent endpoint updates get then discarded due to not finding
the corresponding service anymore. Most notably, this bug is triggered if
the service in the local cluster has no selector, as in turn it is not
associated with any endpointslice, and the service in the remote cluster,
initially marked as global, gets deleted (or no longer shared). At that
point, even if the remote service gets recreated (or marked global again),
the remote endpoints are not merged anymore.

Let's get this fixed by not deleting the service entry from the service
cache in this case, as it is always removed upon actual deletion of the
local service object. Let's also add a unit test to cover this specific
scenario, to validate that the fix works as expected (the test does fail
without the fix).

This bug can be also reproduced with:

  make kind-clustermesh && make kind-clustermesh-images && make kind-install-cilium-clustermesh

  kubectl --context kind-clustermesh1 create deploy podinfo --image=stefanprodan/podinfo --replicas=1
  kubectl --context kind-clustermesh1 expose deploy podinfo --port 80 --target-port 9898
  kubectl --context kind-clustermesh1 annotate svc podinfo service.cilium.io/global=true

  # Create a service without a selector
  cat <<EOF | kubectl --context kind-clustermesh2 create -f -
  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.cilium.io/global: "true"
    name: podinfo
    namespace: default
  spec:
    ports:
    - port: 80
      targetPort: 9898
  EOF

  # Marking the remote service as not global triggers the deletion of
  # the service entry, and marking it as global again does not add back
  # the remote endpoint entries.
  kubectl --context kind-clustermesh1 annotate svc podinfo service.cilium.io/global-
  kubectl --context kind-clustermesh1 annotate svc podinfo service.cilium.io/global=true

Fixes: b7d58c1 ("service-cache: cleanup external endpoints and services on delete")
Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>