Motivation

This PR implements Kubernetes service discovery for SGL Router, allowing it to automatically discover and manage worker pods based on label selectors. The router can dynamically add worker URLs when healthy pods are discovered and remove them when pods are unhealthy or deleted.
Key Features

• Automatic Worker Discovery: Auto-detects worker pods with matching labels
• Dynamic Worker Management: Adds/removes pods based on health status
• Namespace Support: Can watch pods in a specific namespace or cluster-wide (in case router becomes a cluster scope ingress gateway)
• Configurable Worker URL Generation: Uses pod IP and configurable port

Modifications

• Created a new service_discovery.rs module for Kubernetes integration
• Added configuration options to the Router struct and constructor
• Added CLI parameters for service discovery configuration
• Used the Kubernetes API to watch pod events in real-time

Usage

python -m sglang_router.launch_router \
    --service-discovery \
    --selector app=sglang-worker \
    --service-discovery-port 8000 \
    --service-discovery-namespace default

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.