We're almost done with the "legacy" IPCache APIs. The biggest remaining holdout is FQDN policy. For good reason: FQDN policy has the most complex lifecycle and many-to-many mappings everywhere.

This issue proposes a new architecture for the FQDN selector system. It aims for a simpler lifecycle on updates and some code removal.

Background

The FQDN policy system has three main components:

• DNSCache: A forward and reverse mapping of name -> ips (along with some expiration information). This is the ultimate source of (name -> IP) mappings as well as the "owner" of IP lifecycle.
• NameManager: A list of currently active FQDN selectors
• SelectorCache: Generates a set of allowed identities for a given selector

Proposal

Right now, identities for FQDN-policy CIDRs are generated directly by the NameManager and passed in to the SelectorCache.

My proposal would be to move identity management fully to the NameManager, and make FQDN selectors just a bag of cidr:xxxx/32 selectors.

1. A FQDN selector is created. It calls NameManager.RegisterForFQDNUpdates() and preloads its cidr set from existing known names
2. A DNS response is intercepted, and - as before - the NameManager is notified:
   • The NameManager updates its internal datastructures, and determines if any changes are necessary. Assuming it is...
   • The NameManager calls ipcache.UpsertMetadata(ip)
   • All affected selectors have the method AddLabels(cidr:ip/32`)
   • If any of the newly selected cidr labels already have an identity, the selector is updated and an incremental update is pushed to the policy engine
   • When the ipcache allocates any new identities, it will directly update the SelectorCache, which now pushes these updates down to the FQDN selectors
3. The TTLs + conntrack run out, and some IPs are removed (this remains unchanged). Again, the NameManager is responsible for propagating this update. In the new flow, the NameManager calls selector.RemoveLabels("cidr:stale_ip/32") which causes the selector to update its set of cached identities.

Blocking behavior on startup

One concern about migrating to the asynchronous APIs is that the ipcache's asynchronous apply loop is blocked until k8s caches are synced. The FQDN proxy right now starts listening quite early - in fact, in newDaemon(), not in startDaemon(). Any DNS responses received between proxy start and k8s cache population will not be able to allocate new identities. This could lead to policy drops if a new address is detected.

I can't immediately see a way around this; CIDRs will eventually have arbitrary labels, so we don't want to insert them before caches are synced or else numeric identities will flap. But delaying their insertion could lead to noticeable policy drops.

An aside: synchronicity

There is an impedance mismatch between the asynchronous ipcache APIs and the DNS proxy. It attempts to ensure all identities are plumbed before returning the DNS response to the client. We need to build in a way to wait for identities to be made available. Additionally, we should not wait when desired identities are already fulfilled. This will require some API changes to the ipcache.

The plan

I expect the sequence of commits to look like this:

1. Move identity management entirely to the NameManager
2. Convert FQDNSelector to a "bag of labels"
3. Refactor SelectorCache internal APIs
4. Return a channel that is closed when the ipcache has applied all pending prefixes
5. Convert the NameManager to the ipcache asynchronous APIs