Skip to content

Conversation

hsalluri259
Copy link

@hsalluri259 hsalluri259 commented May 2, 2025

Cilium currently do not support releasing IPPrefixes due to which IP starvation is happening in our Kubernetes clusters. We have to manually delete a node in order to free unused IPPrefixes. With this fix, operator will release those unused IPPrefixes to reassign them again in AWS ENI. Added no-op stub implementation in azure and alibabacloud to satisfy NodeOperations interface.

Fixes: #32209, #39904

Without this fix, delegated prefixes won’t be released until their nodes are deleted, resulting in wasted IP space.

Screenshot 2025-05-02 at 12 03 43 PM

With the fix in place, IP Prefixes are released and can be reused again.

Screenshot 2025-05-02 at 12 21 05 PM

Non goals :

  • IPv6 prefix support
Support IPPrefix unassignment in order to reuse those IPPrefixes and prevent IP starvation.
This would require cilium-operator's AWS IAM role update to add "ec2:DescribeRouteTables" permissions.  

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 2, 2025
@hsalluri259 hsalluri259 marked this pull request as ready for review May 2, 2025 20:56
@hsalluri259 hsalluri259 requested review from a team as code owners May 2, 2025 20:56
@hsalluri259 hsalluri259 requested a review from doniacld May 2, 2025 20:56
@github-actions github-actions bot added the kind/community-contribution This was a contribution made by a community member. label May 2, 2025
@doniacld doniacld added kind/bug This is a bug in the Cilium logic. release-note/bug This PR fixes an issue in a previous release of Cilium. labels May 5, 2025
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 5, 2025
@doniacld doniacld added area/azure Impacts Azure based IPAM. dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. labels May 5, 2025
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label May 5, 2025
@doniacld doniacld added area/eni Impacts ENI based IPAM. area/alibaba Impacts Alibaba based IPAM. labels May 5, 2025
Copy link
Contributor

@doniacld doniacld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit remark and a question. Otherwise LGTM.
NB: I think that the commit description is not well formatted and will exceed chars limit per line.

I will have a look in order to reopen the issue

@doniacld doniacld added release-note/minor This PR changes functionality that users may find relevant to operating Cilium. and removed kind/bug This is a bug in the Cilium logic. release-note/bug This PR fixes an issue in a previous release of Cilium. labels May 5, 2025
@hsalluri259 hsalluri259 force-pushed the unassign_aws_eni_ip_prefixes branch from 62d67c2 to 5580b40 Compare May 5, 2025 16:32
@maintainer-s-little-helper
Copy link

Commit c72021a does not match "(?m)^Signed-off-by:".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label May 5, 2025
@hsalluri259 hsalluri259 force-pushed the unassign_aws_eni_ip_prefixes branch from c72021a to 8f41fb5 Compare May 5, 2025 17:24
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label May 5, 2025
@hsalluri259 hsalluri259 requested review from doniacld and PavelGloba May 5, 2025 18:19
@gandro gandro requested review from christarazi and removed request for a team July 2, 2025 07:07
@gandro
Copy link
Member

gandro commented Jul 2, 2025

This needs an ACK from alibabacloud and azure

@maintainer-s-little-helper maintainer-s-little-helper bot removed the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jul 2, 2025
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jul 3, 2025
auto-merge was automatically disabled July 3, 2025 02:11

Head branch was pushed to by a user without write access

@maintainer-s-little-helper
Copy link

Commit b49935c does not match "(?m)^Signed-off-by:".

Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Jul 3, 2025
@hsalluri259 hsalluri259 force-pushed the unassign_aws_eni_ip_prefixes branch from b49935c to 3e8b7e2 Compare July 3, 2025 02:12
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-sign-off The author needs to add signoff to their commits before merge. label Jul 3, 2025
@hsalluri259
Copy link
Author

@gandro I restored these single-line comments as we didn't change the ExcessIPs calculation.
b49935c

@gandro
Copy link
Member

gandro commented Jul 3, 2025

/test

@joestringer
Copy link
Member

Please rebase your PR against the tip of the main branch. Some of the test failures appear to be caused by the PR being quite out-of-date vs the tests that are run (which are pulled from the latest main branch).

@joestringer joestringer added dont-merge/needs-rebase This PR needs to be rebased because it has merge conflicts. and removed ready-to-merge This PR has passed all tests and received consensus from code owners to merge. labels Jul 3, 2025
@hsalluri259 hsalluri259 force-pushed the unassign_aws_eni_ip_prefixes branch from 3e8b7e2 to 05bf3fa Compare July 3, 2025 20:25
@joestringer
Copy link
Member

/test

@joestringer joestringer removed the dont-merge/needs-rebase This PR needs to be rebased because it has merge conflicts. label Jul 3, 2025
Cilium currently does not support releasing IPPrefixes and secondary
IPs when IPPrefix delegation enabled due to which IP starvation is
happening in our Kubernetes clusters. We have to manually delete a
node in order to free unused IPPrefixes.

With this fix, the operator will release those unused IPPrefixes and
secondary IPs to reassign them again in AWS ENI.

Added a no-op stub implementation in Alibaba and Azure Clouds to satisfy the
NodeOperations interface.

Fixes: cilium#32209, cilium#39904

Signed-off-by: Harish Salluri <hsalluri259@gmail.com>
@hsalluri259 hsalluri259 force-pushed the unassign_aws_eni_ip_prefixes branch from 05bf3fa to 342f32c Compare July 6, 2025 20:32
@hsalluri259
Copy link
Author

/test
I rebased it again as I saw couple of tests failing. Please run the tests again.

@gandro
Copy link
Member

gandro commented Jul 7, 2025

/test

@gandro gandro enabled auto-merge July 7, 2025 11:43
@gandro gandro added this pull request to the merge queue Jul 7, 2025
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jul 7, 2025
Merged via the queue into cilium:main with commit 886a511 Jul 7, 2025
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/alibaba Impacts Alibaba based IPAM. area/azure Impacts Azure based IPAM. area/eni Impacts ENI based IPAM. kind/community-contribution This was a contribution made by a community member. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/minor This PR changes functionality that users may find relevant to operating Cilium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cilium operator doesn't release excess IPv4 CIDR block in AWS ENI when prefix delegation is enabled
7 participants