Skip to content

Conversation

daveads
Copy link
Contributor

@daveads daveads commented Jun 15, 2025

NATS Connection Exhaustion: Maximum Connections Exceeded

Detects high-severity NATS server failures when the configured maximum connection limit is exceeded, preventing new clients from establishing connections. This represents a critical messaging infrastructure failure that can cause cascading outages across distributed systems relying on NATS for inter-service communication and real-time data streaming.

This rule helps identify problems such as:

  • NATS server max_connections setting too low for current workload demands
  • Auto-scaling events creating simultaneous connection burst attempts that overwhelm the server
  • Connection pool misconfigurations in client applications creating excessive connections
  • Service restarts causing mass reconnection attempts within short time windows
  • Connection leaks from applications not properly closing NATS connections after use
  • Kubernetes pod restarts causing simultaneous reconnection storms during deployments
  • Load testing or traffic spikes overwhelming NATS connection limits and blocking legitimate traffic

Reproduction: Docker-based setup with NATS server (3-connection limit) and 6 client containers to reliably trigger connection exhaustion scenarios.

/claim #77
/close #77

Reproducible test setup (Maintainers invited) : nats connection

Live CRE Link : CRE playground

@daveads
Copy link
Contributor Author

daveads commented Jun 15, 2025

review_nat.mp4

@tonymeehan tonymeehan merged commit 2965531 into prequel-dev:main Jun 16, 2025
2 checks passed
@tonymeehan
Copy link
Contributor

Great rule. This was a good problem to find and fix since it's very specific and rarely benign. Good use of negates to avoid FPs as well.

@daveads
Copy link
Contributor Author

daveads commented Jun 18, 2025

Great rule. This was a good problem to find and fix since it's very specific and rarely benign. Good use of negates to avoid FPs as well.

Thanks...

@daveads
Copy link
Contributor Author

daveads commented Jun 18, 2025

@tonymeehan when do i get awarded the bounty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Multiple Winners] NATS: Reproduce A High-Severity Failure & Write a Detection Rule [Submit by June 15 11:59 pm ET]
2 participants