Skip to content

Conversation

nberlee
Copy link
Contributor

@nberlee nberlee commented Jul 10, 2023

This Pull Request introduces the 'EnableHealthCheckLoadBalancerIP' flag, which exposes the healthCheckNodePort on the LoadBalancerIP (LB IP). This addition aims to resolve an issue where clusters hosted on GCP in KubeProxyReplacement mode, using a loadbalancer-service with externalTrafficPolicy: Local, were triggering failures in GCP health checks. This, in turn, was causing ingress traffic to be distributed among all Kubernetes nodes, which is not compatible with the Local traffic policy.

Specifically, GCP's Load Balancer performs the health checks using the LB IP as the destination IP rather than the node IP. While the cloud-provider-gcp sets up these healthchecks based on the services' spec.healthCheckNodePort which is accessible by Cilium on the nodeIP, it doesn't acknowledge the GCP LB healthcheck packets because Cilium's HealthServer binds to local IPs and doesn't have forwarding rules for the LB IPs.

Please ensure your pull request adheres to the following guidelines:

  • For first time contributors, read Submitting a pull request
  • All code is covered by unit and/or runtime tests where feasible.
  • All commits contain a well written commit description including a title,
    description and a Fixes: #XXX line if the commit addresses a particular
    GitHub issue.
  • If your commit description contains a Fixes: <commit-id> tag, then
    please add the commit author[s] as reviewer[s] to this issue.
  • All commits are signed off. See the section Developer’s Certificate of Origin
  • Provide a title or release-note blurb suitable for the release notes.
  • Are you a user of Cilium? Please add yourself to the Users doc
  • Thanks for contributing!

Fixes: #26727

Added the EnableHealthCheckLoadBalancerIP flag to address health checks on LoadBalancerIP in Google Cloud Platform using KubeProxyReplacement.

@nberlee nberlee requested review from a team as code owners July 10, 2023 09:51
@nberlee nberlee requested review from ysksuzuki, thorn3r and asauber July 10, 2023 09:51
@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jul 10, 2023
@nberlee nberlee requested review from joamaki and kaworu July 10, 2023 09:51
@github-actions github-actions bot added the kind/community-contribution This was a contribution made by a community member. label Jul 10, 2023
@nberlee nberlee requested a review from a team as a code owner July 10, 2023 11:25
@nberlee nberlee requested a review from a user July 10, 2023 11:25
Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs (autogenerated) ok

Copy link
Member

@kaworu kaworu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @nberlee, I left a cosmetic comment but other than that Helm changes LGTM.

@kaworu kaworu added kind/feature This introduces new functionality. release-note/minor This PR changes functionality that users may find relevant to operating Cilium. area/loadbalancing Impacts load-balancing and Kubernetes service implementations labels Jul 10, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jul 10, 2023
Copy link
Member

@kaworu kaworu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update @nberlee, Helm changes LGTM now.

Copy link
Contributor

@joamaki joamaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few nits, but other than that implementation looks good to me.
I would however like to see a test for this. It should be fairly straightforward to add to
pkg/service/service_test.go. TestHealthCheckNodePort is likely a good starting point.
Remember to restore option.Config.EnableHealthCheckLoadBalancerIP after the test case.

@nberlee
Copy link
Contributor Author

nberlee commented Jul 17, 2023

I would however like to see a test for this. It should be fairly straightforward to ...

@joamaki I agree that a test would be nice, but its not straightforward for me when it come to mocking node.GetIPv4() / node.GetIPv6() not sure what the right approach is there.

@brb brb added the dont-merge/needs-rebase This PR needs to be rebased because it has merge conflicts. label Jul 18, 2023
@nberlee nberlee requested a review from joamaki July 18, 2023 15:50
@nberlee
Copy link
Contributor Author

nberlee commented Jul 25, 2023

@joamaki friendly reminder :)
I've added the requested tests and have changed the requested changes

Copy link
Contributor

@thorn3r thorn3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agent changes lgtm 👍 nice work!

@nberlee
Copy link
Contributor Author

nberlee commented Jul 26, 2023

Great that the PR is approved, could anymody issue a
/test
Please? also I do not think the needs-rebase label is warranted anymore. (rebased yesterday again)

@youngnick youngnick removed the dont-merge/needs-rebase This PR needs to be rebased because it has merge conflicts. label Jul 28, 2023
@youngnick
Copy link
Contributor

/test

This commit introduces the 'EnableHealthCheckLoadBalancerIP', which exposes
the healthCheckNodePort on the LoadBalancerIP. The healthcheck server doesn't
bind the healthCheckNodePort on the LoadBalancerIP (LBip). Instead, Google
Cloud Platform's Load Balancer performs the health checks using the LB IP as
the destination IP rather than the node IP.

To facilitate this change, 'EnableHealthCheckLoadBalancerIP' adds service
specifically for the healthcheck port. The creation and removal of this
additional service are coupled to the primary service that requires health
checking.

Fixes: #26727

Signed-off-by: Nico Berlee <nico.berlee@on2it.net>
@nberlee
Copy link
Contributor Author

nberlee commented Jul 31, 2023

/test

@pchaigno pchaigno merged commit 7a062e3 into cilium:main Aug 2, 2023
brb added a commit that referenced this pull request Jun 3, 2025
Create a LoadBalancer service for HC only if
EnableHealthCheckLoadBalancerIP is set. More ctx about the latter -
#26728.

Signed-off-by: Martynas Pumputis <martynas@isovalent.com>
brb added a commit that referenced this pull request Jun 3, 2025
Create a LoadBalancer service for HC only if
EnableHealthCheckLoadBalancerIP is set. More ctx about the latter -
#26728.

Signed-off-by: Martynas Pumputis <martynas@isovalent.com>
brb added a commit that referenced this pull request Jun 3, 2025
Create a LoadBalancer service for HC only if
EnableHealthCheckLoadBalancerIP is set. More ctx about the latter -
#26728.

Signed-off-by: Martynas Pumputis <martynas@isovalent.com>
brb added a commit that referenced this pull request Jun 13, 2025
Create a LoadBalancer service for HC only if
EnableHealthCheckLoadBalancerIP is set. More ctx about the latter -
#26728.

Signed-off-by: Martynas Pumputis <martynas@isovalent.com>
brb added a commit that referenced this pull request Jun 16, 2025
Create a LoadBalancer service for HC only if
EnableHealthCheckLoadBalancerIP is set. More ctx about the latter -
#26728.

Signed-off-by: Martynas Pumputis <martynas@isovalent.com>
brb added a commit that referenced this pull request Jun 16, 2025
Create a LoadBalancer service for HC only if
EnableHealthCheckLoadBalancerIP is set. More ctx about the latter -
#26728.

Signed-off-by: Martynas Pumputis <martynas@isovalent.com>
brb added a commit that referenced this pull request Jun 16, 2025
Create a LoadBalancer service for HC only if
EnableHealthCheckLoadBalancerIP is set. More ctx about the latter -
#26728.

Signed-off-by: Martynas Pumputis <martynas@isovalent.com>
harsimran-pabla pushed a commit to harsimran-pabla/cilium that referenced this pull request Jun 17, 2025
Create a LoadBalancer service for HC only if
EnableHealthCheckLoadBalancerIP is set. More ctx about the latter -
cilium#26728.

Signed-off-by: Martynas Pumputis <martynas@isovalent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/loadbalancing Impacts load-balancing and Kubernetes service implementations kind/community-contribution This was a contribution made by a community member. kind/feature This introduces new functionality. release-note/minor This PR changes functionality that users may find relevant to operating Cilium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GCP healthchecks fail in KubeProxyReplacement mode
8 participants