Skip to content

Fix flaky test TestListAutoscalerAllocated #4130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Mar 21, 2025
Merged

Fix flaky test TestListAutoscalerAllocated #4130

merged 2 commits into from
Mar 21, 2025

Conversation

igooch
Copy link
Collaborator

@igooch igooch commented Mar 20, 2025 •
edited
Loading

What type of PR is this?

/kind hotfix

What this PR does / Why we need it:

Which issue(s) this PR fixes:

Quick fix for flaky TestListAutoscalerAllocated Scale Up Buffer Percent test.

Also adds in the fix for the broken link #4129

Special notes for your reviewer:

@github-actions github-actions bot added the kind/hotfix Hotfixes for issues against release label Mar 20, 2025
@agones-bot
Copy link
Collaborator

Build Failed 😭

Build Id: b6338e34-8f39-46a9-97a3-4698b002c78b

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Failed 😭

Build Id: ff4d4130-2c07-42b0-a562-5c8e81c9eb39

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@igooch
Copy link
Collaborator Author

igooch commented Mar 20, 2025

Current issue is with test/e2e TestAllocatorWithSelectors, test/e2e TestAllocatorWithDeprecatedRequired and test/e2e TestRestAllocatorWithDeprecatedRequired on cluster gke-autopilot-e2e-test-cluster-1-32.

Fleet replicas stuck in Starting:

me@me:~/agones/test/e2e$ k get fleet -n 1742489291
NAME                    SCHEDULING   DESIRED   CURRENT   ALLOCATED   READY   AGE
simple-fleet-1.0cgf2k   Packed       5         5         0           0       151m
simple-fleet-1.0dzjpp   Packed       5         5         0           0       141m
simple-fleet-1.0sb7gz   Packed       5         5         0           0       161m
me@me:~/agones/test/e2e$ k get gs -n 1742489291
NAME                                STATE      ADDRESS   PORT   NODE   AGE
simple-fleet-1.0cgf2k-rn5pm-4jbk7   Starting                           151m
simple-fleet-1.0cgf2k-rn5pm-4nwpw   Starting                           151m
simple-fleet-1.0cgf2k-rn5pm-dprw2   Starting                           151m
simple-fleet-1.0cgf2k-rn5pm-dpwcw   Starting                           151m
simple-fleet-1.0cgf2k-rn5pm-kbhgz   Starting                           151m
simple-fleet-1.0dzjpp-lp8tn-7zmfw   Starting                           141m
simple-fleet-1.0dzjpp-lp8tn-bhl8c   Starting                           141m
simple-fleet-1.0dzjpp-lp8tn-gc74l   Starting                           141m
simple-fleet-1.0dzjpp-lp8tn-mg7cc   Starting                           141m
simple-fleet-1.0dzjpp-lp8tn-mhkxb   Starting                           141m
simple-fleet-1.0sb7gz-8hrkz-dcrzq   Starting                           161m
simple-fleet-1.0sb7gz-8hrkz-nwnbk   Starting                           161m
simple-fleet-1.0sb7gz-8hrkz-qkj7b   Starting                           161m
simple-fleet-1.0sb7gz-8hrkz-wt84j   Starting                           161m
simple-fleet-1.0sb7gz-8hrkz-zrzjn   Starting                           161m

Event logs:

me@me:~/agones/test/e2e$ k get events -n 1742489291
LAST SEEN   TYPE      REASON                   OBJECT                                  MESSAGE
41s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0cgf2k-rn5pm-4jbk7   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "14fad3009efee67dd482cebc1280b189adcecf22c84bd9435682365c34c68e83": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
116s        Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0cgf2k-rn5pm-4nwpw   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5f09e8ef23e7ffb202a039603bb657e310467b55d4060dddad259e3a07c9c5a9": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
40s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0cgf2k-rn5pm-dprw2   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7247214fa1057490e2d06e82dcee134744e0f321fecc048bedfb4cb0b40e8bc6": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
45s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0cgf2k-rn5pm-dpwcw   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "aca78be4c27fc612f61d3d2f28106b434d02ec798272b5bed44ac53a905c8476": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
2m9s        Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0cgf2k-rn5pm-kbhgz   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f624ebdc0fc77a195b069b116935c95691f33df9141fde79ca21d445edbe63ac": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
68s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0dzjpp-lp8tn-7zmfw   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3c1b482cfc142c2ff407ed8db967c37ff2a4d4ccc6765af956de02018fb78d0f": plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests
89s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0dzjpp-lp8tn-bhl8c   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "99cc4a06eda3e8ce3304b4665585d166d82bfece94ea3c037e8f649a38331830": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
91s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0dzjpp-lp8tn-gc74l   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e67a20a9bcdab9a6bc9ab30f29cc6c8e7c47a5a5c3be16254744b0638d9758c2": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
61s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0dzjpp-lp8tn-mg7cc   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9f611b2b018b917375a34209a53d87c89f8abf51c49863117b9e74cd5e92530e": plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests
80s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0dzjpp-lp8tn-mhkxb   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dd8853c198a5982219ada6576a3684b9c8ef5935fbe51bcd1cff99464e4ad7f1": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
67s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0sb7gz-8hrkz-dcrzq   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "315a2c65e219dfd1643ec15989bdd16d8be2e0c8f49c08216cc800c2e6009add": plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests
59s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0sb7gz-8hrkz-nwnbk   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "88daf4da02d3caeb2721b83aad7d6211969bc549bd7875c07302c7970d730b6b": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded
62s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0sb7gz-8hrkz-qkj7b   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "948fd12f347bbbb13bc44f6d843239aa1e156b9e40df0c65fb8593456c5acfb5": plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests
98s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0sb7gz-8hrkz-wt84j   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0089f28c5e8b66c6c65a470d6b2f6689d8fc7b3ef99d9862f2f04032a9a17f3b": plugin type="cilium-cni" failed (add): unable to create endpoint: [PUT /endpoint/{id}][429] putEndpointIdTooManyRequests
90s         Warning   FailedCreatePodSandBox   pod/simple-fleet-1.0sb7gz-8hrkz-zrzjn   (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8673f88045d62b0c1d72bd6fb78d8a2cb2b6b7eede0b31b208a517139e0164a1": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded

@agones-bot
Copy link
Collaborator

Build Failed 😭

Build Id: 8585cce7-4379-4d89-a6f4-09902ae9dc84

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Succeeded 🥳

Build Id: 19b3b7e3-3e9b-496c-9b30-8c2a1fe43ab7

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4130/head:pr_4130 && git checkout pr_4130
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.48.0-dev-2182931

@igooch
Copy link
Collaborator Author

igooch commented Mar 20, 2025

Current issue is with test/e2e TestAllocatorWithSelectors, test/e2e TestAllocatorWithDeprecatedRequired and test/e2e TestRestAllocatorWithDeprecatedRequired on cluster gke-autopilot-e2e-test-cluster-1-32.

Seems to have been resolved by upgrading the cluster patch version, which does a rolling restart of the cluster.

@igooch igooch requested a review from peterzhongyi March 20, 2025 23:03
peterzhongyi
peterzhongyi reviewed Mar 20, 2025
View reviewed changes
peterzhongyi
peterzhongyi approved these changes Mar 21, 2025
View reviewed changes
Copy link
Collaborator

@peterzhongyi peterzhongyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@peterzhongyi peterzhongyi merged commit c7e8e81 into googleforgames:main Mar 21, 2025
4 checks passed
@0xaravindh 0xaravindh mentioned this pull request Mar 21, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peterzhongyi peterzhongyi peterzhongyi approved these changes

Assignees
No one assigned
Labels
kind/hotfix Hotfixes for issues against release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@igooch @agones-bot @peterzhongyi