Skip to content

sysdump: Collect crashed pod logs in cilium-test namespaces #35612

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 4, 2024
Merged

sysdump: Collect crashed pod logs in cilium-test namespaces #35612

merged 1 commit into from
Nov 4, 2024

Conversation

jschwinger233
Copy link
Member

@jschwinger233 jschwinger233 commented Oct 29, 2024 •
edited
Loading

With this patch, cilium-cli sysdump collects pod logs with CrashLoopBackOff status in cilium-test* namespaces.

An example of use case is https://github.com/cilium/cilium/actions/runs/11574170010 which fails to run echo pods for connectivity test. Now that sysdump can collect logs of echo pods, we can easily find the cause by checking their logs:

# cat logs-echo-same-node-6c868b545b-l9hbr-echo-same-node-20241029-125929.log 

2024-10-29T12:57:35.817979304Z Error: EMFILE: too many open files, watch '/'
2024-10-29T12:57:35.817994894Z     at FSWatcher.<computed> (node:internal/fs/watchers:247:19)
2024-10-29T12:57:35.817997989Z     at Object.watch (node:fs:2469:36)
2024-10-29T12:57:35.818000935Z     at /usr/local/lib/node_modules/json-server/lib/cli/run.js:163:10 {
2024-10-29T12:57:35.818003890Z   errno: -24,
2024-10-29T12:57:35.818007367Z   syscall: 'watch',
2024-10-29T12:57:35.818010582Z   code: 'EMFILE',
2024-10-29T12:57:35.818014320Z   path: '/',
2024-10-29T12:57:35.818017526Z   filename: '/'
2024-10-29T12:57:35.818020291Z }

@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Oct 29, 2024
@github-actions github-actions bot added the cilium-cli This PR contains changes related with cilium-cli label Oct 29, 2024
@jschwinger233
Copy link
Member Author

/ci-e2e-upgrade

@jschwinger233 jschwinger233 force-pushed the pr/gray/collect-crashed-pods branch from 729b14b to 0b7613b Compare October 29, 2024 12:42
@jschwinger233
Copy link
Member Author

/ci-e2e-upgrade

1 similar comment
@jschwinger233
Copy link
Member Author

/ci-e2e-upgrade

@jschwinger233 jschwinger233 changed the title Pr/gray/collect crashed pods sysdump: Collect crashed pod logs in cilium-test namespaces Oct 29, 2024
@jschwinger233 jschwinger233 added the release-note/misc This PR makes changes that have no direct user impact. label Oct 29, 2024
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Oct 29, 2024
@jschwinger233 jschwinger233 force-pushed the pr/gray/collect-crashed-pods branch from 0b7613b to 4c4abdc Compare October 29, 2024 13:11
@jschwinger233
Copy link
Member Author

/test

@jschwinger233 jschwinger233 marked this pull request as ready for review October 29, 2024 13:12
@jschwinger233 jschwinger233 requested a review from a team as a code owner October 29, 2024 13:12
derailed
derailed requested changes Oct 29, 2024
View reviewed changes
Copy link
Contributor

@derailed derailed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jschwinger233 Thank you for this update!

@jschwinger233
sysdump: Collect crashed cilium-test pod logs
f64174d 
Signed-off-by: gray <gray.liang@isovalent.com>
@jschwinger233 jschwinger233 force-pushed the pr/gray/collect-crashed-pods branch from 4c4abdc to f64174d Compare October 30, 2024 03:11
@jschwinger233
Copy link
Member Author

/test

@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Nov 4, 2024
@pchaigno pchaigno added this pull request to the merge queue Nov 4, 2024
Merged via the queue into main with commit 76e3b5e Nov 4, 2024
209 checks passed
@pchaigno pchaigno deleted the pr/gray/collect-crashed-pods branch November 4, 2024 20:15
@github-actions github-actions bot mentioned this pull request Dec 9, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@derailed derailed derailed approved these changes

Assignees
No one assigned
Labels
cilium-cli This PR contains changes related with cilium-cli ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jschwinger233 @derailed @pchaigno