fix aws-cni conformance test #20049
Merged
fix aws-cni conformance test #20049
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
christarazi
approved these changes
Jun 1, 2022
pchaigno
requested changes
Jun 2, 2022
pchaigno
requested changes
Jun 2, 2022
| @@ -155,6 +155,8 @@ jobs: | |||
| --helm-set=hubble.relay.image.tag=${SHA} \ | |||
| --helm-set=enableIPv4Masquerade=false \ | |||
| --helm-set=cni.chainingMode=aws-cni \ | |||
| --helm-set=eni.enabled=false \ | |||
| --helm-set=ipam.mode=cluster-pool \ | |||
There was a problem hiding this comment.
Can we have a comment here to explain why setting what looks like the default value is necessary?
Commit 2d7af3a (".github: add support for cilium-cli in aws-cni conformance tests") changed the AWS-CNI workflow to install Cilium using the CLI instead of Helm. All of the Helm flags were passed through the CLI using --helm-set. However, the CLI also does it's own magic and isn't aware that we want to install Cilium in chaining mode. Therefore, it detects EKS and incorrectly sets IPAM mode to ENI. As a result, Cilium attempts to setup the IP rules and routes for new pods. It seems to fail in most cases—maybe because AWS-CNI already setup the state—but sometimes succeeds. When it succeeds, pods end up with an egress rule pointing to a non-existing ip route table. This usually surfaces in the workflow runs as a DNS failure: the client pod fails to egress to the DNS backend on a remote node. In rarer cases, it surfaces as a failing connectivity test for some of the pods. This commit fixes it by overwriting some of the helm flags set automatically by Cilium-cli namely "eni.enabled=false" and "ipam=cluster-pool". The longer-term fix would be to support chaining mode in the CLI. Reported-by: Paul Chaignon <paul@cilium.io> Signed-off-by: André Martins <andre@cilium.io>
|
@aanm You know you can retrigger a run of the workflow in the top right corner of the previous run, without having to close and open the PR, right? |
Where's the fun in that? 😄 |
|
That's a different flake. We should file an issue for it. Not sure how frequent, but if it is frequent enough, then maybe we shouldn't reenable. It's a very weird one: policy denied drops on egress for pod-to-pod traffic on the same node, with correct identities, and a correct policy map at the time we collect the sysdump. |
This was referenced Jun 6, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Commit 2d7af3a (".github: add support for cilium-cli in aws-cni
conformance tests") changed the AWS-CNI workflow to install Cilium using
the CLI instead of Helm. All of the Helm flags were passed through the
CLI using --helm-set.
However, the CLI also does it's own magic and isn't aware that we want
to install Cilium in chaining mode. Therefore, it detects EKS and
incorrectly sets IPAM mode to ENI.
As a result, Cilium attempts to setup the IP rules and routes for new
pods. It seems to fail in most cases—maybe because AWS-CNI already setup
the state—but sometimes succeeds. When it succeeds, pods end up with an
egress rule pointing to a non-existing ip route table.
This usually surfaces in the workflow runs as a DNS failure: the client
pod fails to egress to the DNS backend on a remote node. In rarer cases,
it surfaces as a failing connectivity test for some of the pods.
This commit fixes it by overwriting some of the helm flags set
automatically by Cilium-cli namely "eni.enabled=false" and
"ipam=cluster-pool".
The longer-term fix would be to support chaining mode in the CLI.
Reported-by: Paul Chaignon paul@cilium.io
Signed-off-by: André Martins andre@cilium.io