[Doc] Added ray-serve llm doc #52832
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can you fix the CI error? In addition, each commit needs to commit with |
1ff1edc to
b49a3e0
Compare
pcmoritz
approved these changes
May 8, 2025
There was a problem hiding this comment.
Thanks a lot for contributing this, it looks good to me. Before merging this, we first need to merge ray-project/kuberay#3517 :)
@kevin85421 Can you drive that PR forward?
kevin85421
reviewed
May 10, 2025
kevin85421
reviewed
May 11, 2025
kevin85421
reviewed
May 11, 2025
added 6 commits
May 12, 2025 20:26
Signed-off-by: DPatel_7 <dpatel@gocommotion.com>
Signed-off-by: DPatel_7 <dpatel@gocommotion.com>
Signed-off-by: DPatel_7 <dpatel@gocommotion.com>
|
made updates. @kevin85421 |
kevin85421
reviewed
May 13, 2025
Signed-off-by: DPatel_7 <dpatel@gocommotion.com>
kevin85421
reviewed
May 13, 2025
There was a problem hiding this comment.
Would you mind verifying whether the YAML (ray-project/kuberay#3517 (review)) still works or not? This doc has removed the step of creating a namespace, but the YAML still uses the namespace.
added 2 commits
May 28, 2025 12:38
eicherseiji
requested changes
Jun 9, 2025
eicherseiji
approved these changes
Jun 9, 2025
dstrodtman
reviewed
Jun 9, 2025
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
|
@kevin85421 Ready to merge when green |
kevin85421
reviewed
Jun 10, 2025
angelinalg
reviewed
Jun 10, 2025
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
kevin85421
approved these changes
Jun 12, 2025
| For additional security, instead of passing the HF Access Token directly as an environment variable, create a Kubernetes Secret containing your Hugging Face access token. Download the Ray Serve LLM service config .yaml file using the following command: | ||
|
|
||
| ```sh | ||
| curl -o ray-service.llm-serve.yaml https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.llm-serve.yaml |
There was a problem hiding this comment.
We can use the release branch https://github.com/ray-project/kuberay/tree/release-1.4.
There was a problem hiding this comment.
Seems like ray-project/kuberay#3517 needs to be cherry-picked
angelinalg
reviewed
Jun 12, 2025
angelinalg
reviewed
Jun 12, 2025
angelinalg
approved these changes
Jun 12, 2025
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>
elliot-barn
pushed a commit
that referenced
this pull request
Jun 18, 2025
Add example of serving a Large Language Model using Ray Serve LLM on Kubernetes Signed-off-by: DPatel_7 <dpatel@gocommotion.com> Signed-off-by: Seiji Eicher <seiji@anyscale.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com> Co-authored-by: DPatel_7 <dpatel@gocommotion.com> Co-authored-by: Seiji Eicher <seiji@anyscale.com> Co-authored-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
elliot-barn
pushed a commit
that referenced
this pull request
Jul 2, 2025
Add example of serving a Large Language Model using Ray Serve LLM on Kubernetes Signed-off-by: DPatel_7 <dpatel@gocommotion.com> Signed-off-by: Seiji Eicher <seiji@anyscale.com> Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com> Co-authored-by: DPatel_7 <dpatel@gocommotion.com> Co-authored-by: Seiji Eicher <seiji@anyscale.com> Co-authored-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Co-authored-by: Kai-Hsun Chen <kaihsun@apache.org> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Add example of serving a Large Language Model using Ray Serve LLM on Kubernetes
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.