Skip to content

Deploy multi-node inference (LWS method) using sglang in a K8s cluster #3624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Feb 18, 2025
Merged

Deploy multi-node inference (LWS method) using sglang in a K8s cluster #3624

merged 8 commits into from
Feb 18, 2025

Conversation

whybeyoung
Copy link
Collaborator

@whybeyoung whybeyoung commented Feb 17, 2025 •
edited
Loading

Motivation

  1. The community lacks a good example of distributed inference in a K8s environment.
  2. The community lacks examples of containerized environments combined with high-speed networks like RoCE.
  3. K8s is the most popular open-source infrastructure platform, but it lacks best practices for integration with sglang, which is one of the most popular recent open-source inference projects.

FrankLeeeee
FrankLeeeee reviewed Feb 17, 2025
View reviewed changes
FrankLeeeee
FrankLeeeee reviewed Feb 17, 2025
View reviewed changes
FrankLeeeee
FrankLeeeee reviewed Feb 17, 2025
View reviewed changes
FrankLeeeee
FrankLeeeee reviewed Feb 17, 2025
View reviewed changes
@whybeyoung whybeyoung mentioned this pull request Feb 17, 2025
Merged
6 tasks
@zhaochenyang20 zhaochenyang20 enabled auto-merge (squash) February 18, 2025 02:06
@zhaochenyang20 zhaochenyang20 merged commit c51dc2c into sgl-project:main Feb 18, 2025
11 of 12 checks passed
kerthcet
kerthcet reviewed Feb 26, 2025
View reviewed changes
docs/references/multi_node_inference_k8s_lws.md
@@ -0,0 +1,339 @@
# Deploying a RoCE Network-Based SGLANG Two-Node Inference Service on a Kubernetes (K8S) Cluster
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @whybeyoung would you like to submit a PR to lws as a RDMA example, I think this values a lot to our users for RDMA-based quick start, if not, I can submit one. Thanks.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! You can do it on your side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhaochenyang20 zhaochenyang20 zhaochenyang20 approved these changes

@FrankLeeeee FrankLeeeee FrankLeeeee approved these changes

+1 more reviewer

@kerthcet kerthcet kerthcet left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@whybeyoung @kerthcet @FrankLeeeee @zhaochenyang20