Skip to content

Conversation

jrife
Copy link
Contributor

@jrife jrife commented Nov 4, 2024

When a node's public key changes, as is the case after a node restart, all other nodes delete the peer entry for the old key and create a new peer entry for the new key. However, the logic introduced by commit 3ebfa4d ("wireguard: Use dummy peer for allowed IP removal") fails to reinitialize the allowed IPs state for the new entry, namely the node IPs.

TestAgent_AllowedIPsRestoration actually did have test coverage for this scenario, but passed with a false positive since k8s2PubKey and k8s2PubKey2 were not different enough it seems. Setting k8s2PubKey2 to a completely different key makes the unit test fail as it should have.

Reinitialize peer inside UpdatePeer() whenever a node's public key changes to trigger a full resync of node and pod IPs for the new key, and ensure that TestAgent_AllowedIPsRestoration properly tests for this scenario.

Testing

  1. Deploy Cilium to a cluster with WireGuard enabled cilium install --set image.override=$MY_REGISTRY/cilium:latest --set image.pullPolicy=Always --set operator.image.override=$MY_REGISTRY/operator-generic:latest --set operator.image.pullPolicy=Always --set operator.image.suffix="" --set encryption.enabled=true --set encryption.type=wireguard --set encryption.nodeEncryption=true --set kubeProxyReplacement=true --set bpf.masquerade=true --set cluster.name=jrife
  2. Reboot one of the worker nodes
  3. On another worker node observe the output of wg show and cilium status for the node that was rebooted.
Before This Fix

wg show shows the public key for node 10.13.106.162 change and does not resync the node IP(s) until an agent restart. The output below is before and after the node in question reboots.

jrife@gke-cilium-10481617798-1-5-ubuntu-big-9eaebb54-l39a:~$ sudo wg
interface: cilium_wg0
  public key: F2QtZjXwljAiRzkIyihCFxkJxGaCkI208RFir166R1s=
  private key: (hidden)
  listening port: 51871
  fwmark: 0x1e00

peer: cxfTnc0InUC2HF7uT3SeqZGWV593BNfE13ApYLVBKiQ=
  endpoint: 10.13.106.162:51871
  allowed ips: 10.13.106.162/32, 10.102.73.179/32, 10.102.73.166/32, 10.102.73.67/32, 10.102.73.91/32, 10.102.73.145/32, 34.162.99.47/32, 10.102.73.7/32
  latest handshake: 1 minute, 31 seconds ago
  transfer: 1.73 MiB received, 1.18 MiB sent
jrife@gke-cilium-10481617798-1-5-ubuntu-big-9eaebb54-l39a:~$ sudo wg
interface: cilium_wg0
  public key: F2QtZjXwljAiRzkIyihCFxkJxGaCkI208RFir166R1s=
  private key: (hidden)
  listening port: 51871
  fwmark: 0x1e00

peer: 2C7QPnKXaw6/d3eAiNZ2YeqAjR+uqKKLAm5VegaklAk=
  endpoint: 10.13.106.162:51871
  allowed ips: 10.102.73.56/32, 10.102.73.155/32, 10.102.73.195/32, 10.102.73.26/32
  latest handshake: 45 seconds ago
  transfer: 3.28 KiB received, 16.46 KiB sent
jrife@gke-cilium-10481617798-1-5-ubuntu-big-9eaebb54-l39a:~$ 

cilium status shows 1/2 nodes are reachable.

jrife@jrife-kvm:~/code/cilium$ kubectl exec -it -n kube-system cilium-zr6xv -- cilium status
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                                                      Ok   Disabled
Kubernetes:                                                   Ok   1.30 (v1.30.5-gke.1014001) [linux/amd64]
Kubernetes APIs:                                              ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:                                         True   [ens4   10.13.106.161 fe80::4001:aff:fe0d:6aa1 (Direct Routing)]
Host firewall:                                                Disabled
SRv6:                                                         Disabled
CNI Chaining:                                                 none
CNI Config file:                                              successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium:                                                       Ok   1.17.0-dev (v1.17.0-dev-3ecac71128)
NodeMonitor:                                                  Listening for events on 8 CPUs with 64x4096 of shared memory
Cilium health daemon:                                         Ok   
IPAM:                                                         IPv4: 10/254 allocated from 10.102.74.0/24, 
IPv4 BIG TCP:                                                 Disabled
IPv6 BIG TCP:                                                 Disabled
BandwidthManager:                                             Disabled
Routing:                                                      Network: Native   Host: BPF
Attach Mode:                                                  Legacy TC
Device Mode:                                                  veth
Masquerading:                                                 BPF   [ens4]   10.102.72.0/21 [IPv4: Enabled, IPv6: Disabled]
Controller Status:                                            71/71 healthy
Proxy Status:                                                 OK, ip 10.102.74.64, 0 redirects active on ports 10000-20000, Envoy: external
Global Identity Range:                                        min 256, max 65535
Hubble:                                                       Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 16.87   Metrics: Disabled
Encryption:                                                   Wireguard       [NodeEncryption: Enabled, cilium_wg0 (Pubkey: F2QtZjXwljAiRzkIyihCFxkJxGaCkI208RFir166R1s=, Port: 51871, Peers: 1)]
Cluster health:                                               1/2 reachable   (2024-11-04T21:28:10Z)
Name                                                          IP              Node   Endpoints
  jrife/gke-cilium-10481617798-1-5-ubuntu-big-9eaebb54-ehtt   10.13.106.162   0/1    0/1
Modules Health:                                               Stopped(0) Degraded(1) OK(68)
After This Fix

wg show shows the public key for node 10.13.106.162 change and a full set of allowed IPs, including the node IPs. The output below is before and after the node in question reboots.

jrife@gke-cilium-10481617798-1-5-ubuntu-big-9eaebb54-l39a:~$ sudo wg
interface: cilium_wg0
  public key: F2QtZjXwljAiRzkIyihCFxkJxGaCkI208RFir166R1s=
  private key: (hidden)
  listening port: 51871
  fwmark: 0x1e00

peer: 2C7QPnKXaw6/d3eAiNZ2YeqAjR+uqKKLAm5VegaklAk=
  endpoint: 10.13.106.162:51871
  allowed ips: 10.13.106.162/32, 10.102.73.155/32, 10.102.73.195/32, 10.102.73.56/32, 10.102.73.26/32, 34.162.99.47/32, 10.102.73.145/32, 10.102.73.7/32
  latest handshake: 10 seconds ago
  transfer: 399.55 KiB received, 289.02 KiB sent
jrife@gke-cilium-10481617798-1-5-ubuntu-big-9eaebb54-l39a:~$ sudo wg
interface: cilium_wg0
  public key: F2QtZjXwljAiRzkIyihCFxkJxGaCkI208RFir166R1s=
  private key: (hidden)
  listening port: 51871
  fwmark: 0x1e00

peer: N9XGxTQV+XoTymShfcv7sHvkQVMzGqiq08NTcQfR01M=
  endpoint: 10.13.106.162:51871
  allowed ips: 34.162.99.47/32, 10.102.73.145/32, 10.102.73.7/32, 10.13.106.162/32, 10.102.73.42/32, 10.102.73.44/32, 10.102.73.47/32, 10.102.73.21/32
  latest handshake: 1 minute, 29 seconds ago
  transfer: 30.56 KiB received, 38.21 KiB sent
jrife@gke-cilium-10481617798-1-5-ubuntu-big-9eaebb54-l39a:~$ 

cilium status shows 2/2 nodes are reachable.

jrife@jrife-kvm:~/code/cilium$ kubectl exec -it -n kube-system cilium-4ldq6 -- cilium status
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                 Ok   Disabled
Kubernetes:              Ok   1.30 (v1.30.5-gke.1014001) [linux/amd64]
Kubernetes APIs:         ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:    True   [ens4   10.13.106.161 fe80::4001:aff:fe0d:6aa1 (Direct Routing)]
Host firewall:           Disabled
SRv6:                    Disabled
CNI Chaining:            none
CNI Config file:         successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium:                  Ok   1.17.0-dev (v1.17.0-dev-5307ea33da)
NodeMonitor:             Listening for events on 8 CPUs with 64x4096 of shared memory
Cilium health daemon:    Ok   
IPAM:                    IPv4: 10/254 allocated from 10.102.74.0/24, 
IPv4 BIG TCP:            Disabled
IPv6 BIG TCP:            Disabled
BandwidthManager:        Disabled
Routing:                 Network: Native   Host: BPF
Attach Mode:             Legacy TC
Device Mode:             veth
Masquerading:            BPF   [ens4]   10.102.72.0/21 [IPv4: Enabled, IPv6: Disabled]
Controller Status:       71/71 healthy
Proxy Status:            OK, ip 10.102.74.64, 0 redirects active on ports 10000-20000, Envoy: external
Global Identity Range:   min 256, max 65535
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 17.03   Metrics: Disabled
Encryption:              Wireguard       [NodeEncryption: Enabled, cilium_wg0 (Pubkey: F2QtZjXwljAiRzkIyihCFxkJxGaCkI208RFir166R1s=, Port: 51871, Peers: 1)]
Cluster health:          2/2 reachable   (2024-11-04T21:47:40Z)
Name                     IP              Node   Endpoints
Modules Health:          Stopped(0) Degraded(2) OK(67)
jrife@jrife-kvm:~/code/cilium$ 

Fixes: #35644
Fixes: #34612
Fixes: 3ebfa4d ("wireguard: Use dummy peer for allowed IP removal")

wireguard: Fix connectivity issues following node reboots.

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Nov 4, 2024
@jrife jrife force-pushed the jrife/wg-restart-fix branch from 5307ea3 to 2a7a694 Compare November 4, 2024 21:51
@jrife
Copy link
Contributor Author

jrife commented Nov 4, 2024

/test

@jschwinger233 jschwinger233 added area/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. release-note/bug This PR fixes an issue in a previous release of Cilium. area/encryption Impacts encryption support such as IPSec, WireGuard, or kTLS. feature/wireguard Relates to Cilium's Wireguard feature labels Nov 5, 2024
@maintainer-s-little-helper maintainer-s-little-helper bot removed dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. labels Nov 5, 2024
@jrife jrife force-pushed the jrife/wg-restart-fix branch 2 times, most recently from 80472cd to 484baae Compare November 5, 2024 16:55
When a node's public key changes, as is the case after a node restart,
all other nodes delete the peer entry for the old key and create a new
peer entry for the new key. However, the logic introduced by commit
3ebfa4d ("wireguard: Use dummy peer for allowed IP removal") fails
to reinitialize the allowed IPs state for the new entry, namely the
node IPs.

TestAgent_AllowedIPsRestoration actually did have test coverage for this
scenario, but passed with a false positive since k8s2PubKey and
k8s2PubKey2 were not different enough it seems. Setting k8s2PubKey2 to a
completely different key makes the unit test fail as it should have.

Reinitialize peer inside UpdatePeer() whenever a node's public key
changes to trigger a full resync of node and pod IPs for the new key,
and ensure that TestAgent_AllowedIPsRestoration properly tests for this
scenario.

Fixes: cilium#35644
Fixes: cilium#34612
Fixes: 3ebfa4d ("wireguard: Use dummy peer for allowed IP removal")

Signed-off-by: Jordan Rife <jrife@google.com>
@jrife jrife force-pushed the jrife/wg-restart-fix branch from 484baae to 39d463f Compare November 5, 2024 17:26
@jrife jrife marked this pull request as ready for review November 5, 2024 18:52
@jrife jrife requested a review from a team as a code owner November 5, 2024 18:52
@jrife jrife requested a review from jschwinger233 November 5, 2024 18:52
@jrife
Copy link
Contributor Author

jrife commented Nov 5, 2024

cc @jschwinger233 @pchaigno

@pchaigno pchaigno enabled auto-merge November 6, 2024 08:05
@pchaigno
Copy link
Member

pchaigno commented Nov 6, 2024

/test

@pchaigno pchaigno added this pull request to the merge queue Nov 6, 2024
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Nov 6, 2024
Merged via the queue into cilium:main with commit 0495279 Nov 6, 2024
68 checks passed
@julianwiedmann
Copy link
Member

Given that #34612 was backported to v1.16, also adding the label here.

@julianwiedmann julianwiedmann added the needs-backport/1.16 This PR / issue needs backporting to the v1.16 branch label Nov 6, 2024
@viktor-kurchenko viktor-kurchenko mentioned this pull request Nov 12, 2024
13 tasks
@viktor-kurchenko viktor-kurchenko added the backport-pending/1.16 The backport for Cilium 1.16.x for this PR is in progress. label Nov 12, 2024
@viktor-kurchenko viktor-kurchenko removed the needs-backport/1.16 This PR / issue needs backporting to the v1.16 branch label Nov 12, 2024
@github-actions github-actions bot added backport-done/1.16 The backport for Cilium 1.16.x for this PR is done. and removed backport-pending/1.16 The backport for Cilium 1.16.x for this PR is in progress. labels Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. area/encryption Impacts encryption support such as IPSec, WireGuard, or kTLS. backport-done/1.16 The backport for Cilium 1.16.x for this PR is done. feature/wireguard Relates to Cilium's Wireguard feature ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/bug This PR fixes an issue in a previous release of Cilium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Wireguard Allowed IPs not propagated correctly after node restart
5 participants