Description

See detailed description of the issue in #35501

This PR ensures that when BPF Masquerading is enabled together with the BPF ip-masq-agent, the Cilium Agent managed datapath ip rules are actually consistent with the ip-masq-agent configuration. The idea is that all nonMasqueradeCIDRs from ip-masq-agent configuration actually need to be propagated down to the datapath to ensure that these CIDRs are not masqueraded.

As mentioned in the issue, the actual datapath ip rule setup is here:

The newly added logic in my PR simply reads the ip-masq-agent config file and returns the non-masqueradable CIDRs in the CNI IP allocation response.

It's somewhat of a hotpath so I made sure that it fails gracefully if something goes wrong. It also makes the best effort to read the CIDRs from memory and not from disk. I also moved the fsnotify Watcher instantiation to the Start() function to avoid its overhead in the codepath. Finally, in order to avoid race conditions with the ip-masq-agent startup and the IP allocation requests (the IPAM Allocator is started before the ip-masq-agent), this code doesn't rely on the ip-masq-agent being in running state. -> This part of the description is obsolete since I ended up doing a refactoring of the ip-masq-agent in #40347 to extract it into a separate Cell.

One small limitation is that the ip rules will not be updated dynamically if the ip-masq-agent configuration changes. The new CIDRs will apply to all newly started pods however existing pods will need to be restarted in order for the new CIDRs to be applied. IMO it's not a big deal since the situation is already much better than what we had previously.

Testing

Before the PR

Taking the same AWS setup as described in my original issue, when we enable masquerading for a given pod, its ip rules look something like this:

$ ip rule | grep 10.130.100.133
111:	from 10.130.100.133 to 10.128.0.0/13 lookup 11
111:	from 10.130.100.133 to 100.112.0.0/13 lookup 11

The to CIDRs are only coming from the VPC CIDRs auto-detected by the Operator. If we enable the ip-masq-agent, the rules stay the same.

After the PR

Once we deploy the PR and then enable ip-masq-agent with the following config:

  config.yaml: |
    nonMasqueradeCIDRs:
    - 10.0.0.0/8
    - 172.16.0.0/12
    - 100.64.0.0/10
    - 192.168.0.0/16
    masqLinkLocal: false

The CIDRs are now properly propagated down to the ip rule level:

$ ip rule | grep 10.112.49.248
111:	from 10.112.49.248 to 10.0.0.0/8 lookup 11
111:	from 10.112.49.248 to 100.64.0.0/10 lookup 11
111:	from 10.112.49.248 to 169.254.0.0/16 lookup 11
111:	from 10.112.49.248 to 172.16.0.0/12 lookup 11
111:	from 10.112.49.248 to 192.168.0.0/16 lookup 11

We can also confirm that cross-VPC traffic is no longer dropped.

Fixes: #35501

ip-masq-agent: Ensure ip rules on the host match the BPF ip-masq-agent configuration in AWS ENI mode. Note that rules are set up once at pod creation and will not be regenerated if the ip-masq-agent configuration changes.