/test

put back in draft, will have to test restarts when endpoints have L7 policies; it is possible that regeneration now hangs in that case as envoy updates are postponed.

Do we need to update cilium-envoy to the latest to pick up cilium/proxy#785?

I know the context from which this comes -- wanting to ensure Envoy reloads ipcache and doesn't replace policies with an empty set -- but it would be good to identify which sorts of churn this does and does not prevent. What does endpoint restoration provide that Envoy consumes?

As endpoints are regenerated, are we going to see policy drops for non-regenerated endpoints? I assume so, since their policy is not yet generated and thus not in the xDS server. But not sure what we can do about that yet.

/test

I know the context from which this comes -- wanting to ensure Envoy reloads ipcache and doesn't replace policies with an empty set -- but it would be good to identify which sorts of churn this does and does not prevent. What does endpoint restoration provide that Envoy consumes?

Endpoint restoration waits for k8s.CacheStatus (initial sync of all k8s resources), and endpoint restoration itself re-creates all endpoint policies (bpf maps and Envoy policies). k8s sync includes CEC/CCEC resources, so that should cover Ingress/GW API. I would expect these (k8s initial sync & endpoint regeneration) to recreate all Envoy listeners, clusters, endpoints, secrets, as well as network policies.

As endpoints are regenerated, are we going to see policy drops for non-regenerated endpoints? I assume so, since their policy is not yet generated and thus not in the xDS server. But not sure what we can do about that yet.

We now delay sending any new resources to Envoy, so all existing (and most new) connections should keep going based on the previous policy, both in bpf datapath as well as in Envoy.

There is a caveat with new dns proxy-dependent connection in the window between endpoint bpf having been regenerated and us waiting on ALL endpoints having been regenerated before sending updated network policies to Envoy. The updated policy would include newly allocated FQDN IDs.

Endpoint restoration waits for k8s.CacheStatus...

@jrajahalme no, that's endpoint regeneration. Restoration happens much earlier, even before the k8s clients have started. Regeneration is what re-creates policies and proceeds only after the daemon has initialized.

Network policies are, indeed, ingested before regeneration, but regeneration is what converts them to Envoy configuration.

Endpoint restoration waits for k8s.CacheStatus...

@jrajahalme no, that's endpoint regeneration. Restoration happens much earlier, even before the k8s clients have started. Regeneration is what re-creates policies and proceeds only after the daemon has initialized.

Network policies are, indeed, ingested before regeneration, but regeneration is what converts them to Envoy configuration.

My read of the following is that initial k8s state is fetched before endpoints are restored:

	if params.Clientset.IsEnabled() {
		// Wait only for certain caches, but not all!
		// (Check Daemon.InitK8sSubsystem() for more info)
		<-params.CacheStatus
	}
	bootstrapStats.k8sInit.End(true)
	d.initRestore(restoredEndpoints, params.EndpointRegenerator)

Endpoints are regenerated during restoration, so whatever happens during endpoint regeneration also happens during endpoint restoration. Finished regenerating restored endpoints is the last log before endpoint restorations is deemed complete.

/test

/test

/test