Skip to content

Add Slurm critical upstream failure detection rule #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 5, 2025
Merged

Add Slurm critical upstream failure detection rule #57

merged 2 commits into from
Jun 5, 2025

Conversation

varshith257
Copy link
Contributor

@varshith257 varshith257 commented Jun 5, 2025 •
edited
Loading

Details

  • Introduced a new rule for detecting when Slurm's accounting daemon (slurmdbd) loses connection to its MySQL database, impacting job scheduling.
  • Added relevant log entries to demonstrate connection issues.
  • Updated categories and tags to include HPC database problems and SLURM-related tags.

Test Environment

Reproducible test setup (Maintainers invited) : Slurm Cluster
Live CRE link: CRE Playground Link
Check here for Sample Logs

Sample Detected Patterns

[2024-01-15T14:22:10] error: mysql_real_connect failed: 2002 Can't connect to MySQL server
[2024-01-15T14:22:15] error: Processing last message from connection 42 (DB connection lost)
[2024-01-15T14:22:25] error: accounting_storage/slurmdbd: Unable to connect to database

Reproduction Steps

# Start test environment
docker-compose up -d

# Simulate failure (MySQL crash)
docker stop mysql_slurm

# Monitor detection (logs update every 5s)
tail -f /var/log/slurmdbd.log

Closes #43
/claim #43

@varshith257
add slurm CRE
4009786 
- Introduced a new rule for detecting when Slurm's accounting daemon (slurmdbd) loses connection to its MySQL database, impacting job scheduling.
- Added relevant log entries to demonstrate connection issues.
- Updated categories and tags to include HPC database problems and SLURM-related tags.
@algora-pbc algora-pbc bot mentioned this pull request Jun 5, 2025
Closed
@tonymeehan tonymeehan self-requested a review June 5, 2025 17:55
@tonymeehan tonymeehan merged commit 279a893 into prequel-dev:main Jun 5, 2025
2 checks passed
@varshith257 varshith257 deleted the cre-slurm branch June 5, 2025 17:57
@asr2003 asr2003 mentioned this pull request Jun 13, 2025
Closed
This was referenced Sep 1, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tonymeehan tonymeehan tonymeehan approved these changes

Assignees
No one assigned
Labels
🙋 Bounty claim
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[New Rule] Slurm: Reproduce A High-Severity Failure & Write a Detection Rule
2 participants
@varshith257 @tonymeehan