[RayService] adapter vllm 0.6.1.post2 #2823

pxp531 · 2025-01-26T12:01:34Z

Why are these changes needed?

vllm is evolving rapidly, with some api changes in versions 0.6 and 0.5. We used version 0.6.1.post2 of vllm on ray in production practice, so I made a change to this fit

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

andrewsykim · 2025-02-03T04:18:58Z

I ran into this bug when testing this PR:

            Deploying app 'llm' failed with exception:
            Traceback (most recent call last):
              File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/serve/_private/application_state.py", line 1049, in build_serve_application
                app = call_app_builder_with_args_if_necessary(import_attr(import_path), args)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^
              File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/_private/utils.py", line 1191, in import_attr
                module = importlib.import_module(module_name)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/home/ray/anaconda3/lib/python3.11/importlib/__init__.py", line 126, in import_module
                return _bootstrap._gcd_import(name[level:], package, level)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
              File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
              File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
              File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
              File "<frozen importlib._bootstrap_external>", line 940, in exec_module
              File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
              File "/tmp/ray/session_2025-02-02_20-09-21_044154_1/runtime_resources/working_dir_files/https_github_com_pxp531_kuberay_archive_adapter_vllm_6x/ray-operator/config/samples/vllm/serve.py", line 149, in <module>
                model = build_app(
                        ^^^^^^^^^^
              File "/tmp/ray/session_2025-02-02_20-09-21_044154_1/runtime_resources/working_dir_files/https_github_com_pxp531_kuberay_archive_adapter_vllm_6x/ray-operator/config/samples/vllm/serve.py", line 124, in build_app
                parsed_args = parse_vllm_args(cli_args)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/tmp/ray/session_2025-02-02_20-09-21_044154_1/runtime_resources/working_dir_files/https_github_com_pxp531_kuberay_archive_adapter_vllm_6x/ray-operator/config/samples/vllm/serve.py", line 103, in parse_vllm_args
                parser = make_arg_parser(parser)
                                         ^^^^^^
            UnboundLocalError: cannot access local variable 'parser' where it is not associated with a value

Can you fix it?

jannikstdl · 2025-02-03T07:51:20Z

I ran into this bug when testing this PR:

            Deploying app 'llm' failed with exception:
            Traceback (most recent call last):
              File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/serve/_private/application_state.py", line 1049, in build_serve_application
                app = call_app_builder_with_args_if_necessary(import_attr(import_path), args)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^
              File "/home/ray/anaconda3/lib/python3.11/site-packages/ray/_private/utils.py", line 1191, in import_attr
                module = importlib.import_module(module_name)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/home/ray/anaconda3/lib/python3.11/importlib/__init__.py", line 126, in import_module
                return _bootstrap._gcd_import(name[level:], package, level)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
              File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
              File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
              File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
              File "<frozen importlib._bootstrap_external>", line 940, in exec_module
              File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
              File "/tmp/ray/session_2025-02-02_20-09-21_044154_1/runtime_resources/working_dir_files/https_github_com_pxp531_kuberay_archive_adapter_vllm_6x/ray-operator/config/samples/vllm/serve.py", line 149, in <module>
                model = build_app(
                        ^^^^^^^^^^
              File "/tmp/ray/session_2025-02-02_20-09-21_044154_1/runtime_resources/working_dir_files/https_github_com_pxp531_kuberay_archive_adapter_vllm_6x/ray-operator/config/samples/vllm/serve.py", line 124, in build_app
                parsed_args = parse_vllm_args(cli_args)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/tmp/ray/session_2025-02-02_20-09-21_044154_1/runtime_resources/working_dir_files/https_github_com_pxp531_kuberay_archive_adapter_vllm_6x/ray-operator/config/samples/vllm/serve.py", line 103, in parse_vllm_args
                parser = make_arg_parser(parser)
                                         ^^^^^^
            UnboundLocalError: cannot access local variable 'parser' where it is not associated with a value

Can you fix it?

Yes, this is the NameError i suggested a fix below

andrewsykim · 2025-02-03T14:56:48Z

Yes, this is the NameError i suggested a fix below

I don't see any suggestions, did you forget to send the review?

ray-operator/config/samples/vllm/serve.py

jannikstdl · 2025-02-05T09:13:20Z

@andrewsykim now should be up

andrewsykim · 2025-02-06T16:17:17Z

Tested the new code:

$ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
      "model": "meta-llama/Meta-Llama-3-8B-Instruct",
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Provide a brief sentence describing the Ray open-source project."}
      ],
      "temperature": 0.7
    }'
Handling connection for 8000
{"id":"chat-0119a1448c9c41c1ae05ed8a70157653","object":"chat.completion","created":1738858572,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Ray is an open-source Python library that provides a high-performance distributed computing framework for scaling machine learning and data science workflows on clusters, cloud, and edge devices.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":32,"total_tokens":65,"completion_tokens":33},"prompt_logprobs":null}

Thanks @pxp531

andrewsykim

just minor comment remaining about num_cpu

ray-operator/config/samples/vllm/serve.py

* [RayService] More envtests that follow the most common scenario in the RayService code path (#2880) Signed-off-by: Rueian <rueiancsie@gmail.com> * [RayService] Remove outdated env tests (#2886) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [RayService] Refactor envtests (#2888) * [docs][ray-operator] fix typo in Golang version (#2893) Project uses Golang 1.22 not 1.20. Signed-off-by: David Xia <david@davidxia.com> * [Fix][kubectl-plugin] Fix no context nil error SIGSEGV in tests (#2892) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [CI] Enable testifylint rule (#2896) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [CI] Enable `testifylint` `error-nil` rule (#2907) * [kubectl-plugin][feat] support specifying number of head GPUs (#2895) when creating a RayCluster with `kubectl ray create cluster NAME --head-gpu N`. Similar to the `--worker-gpu` switch. Signed-off-by: David Xia <david@davidxia.com> * Use webhook.CustomValidator instead of deprecated webhook.Validator. (#2803) * Use webhook.CustomValidator instead of deprecated webhook.Validator. * Move RayClusterWebhook to pkg/webhook/v1. * [kubectl-plugin] update context error messages (#2891) to tell user they can use `--context` to set the K8s context. Add tests Signed-off-by: David Xia <david@davidxia.com> Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [Fix][kubectl-plugin] make tests use a temporary kube config (#2894) file that's not at the default path. Right now tests fail if there's a K8s current context set with a command like `kubectl config use-context my-context`. This change allows tests to pass regardless of current context. Signed-off-by: David Xia <david@davidxia.com> * [CI] Enable testifylint formatter rule (#2915) Signed-off-by: 400Ping <43886578+400Ping@users.noreply.github.com> * [CI] Enable `testifylint` `empty` rule (#2908) * [CI] Enable testifylint bool-compare rule (#2911) Signed-off-by: 400Ping <43886578+400Ping@users.noreply.github.com> * [CI] Auto download golang tools in pre-commit (#2917) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [RayService] a safeguard for preventing overriding the pending cluster during a upgrade (#2887) Signed-off-by: Rueian <rueiancsie@gmail.com> * [RayService] Refactor unit tests for ShouldPrepareNewCluster (#2928) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [chore][kubectl-plugin] use consistent capitalization (#2922) in comments and console messages. * capitalize "Ray" when used as a proper noun * capitalize Ray K8s CRDs like "RayJob" and "RayCluster" * capitalize acronyms like "YAML" * fix some minor typos No functional changes. Signed-off-by: David Xia <david@davidxia.com> * [CI] Enable `testifylint` `require-error` rule (#2909) * [kubectl-plugin] support general `kubectl` switches like `--context` (#2883) * [CI] Fix lint error (require-error) (#2931) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [CI] Enable `testifylint` `float-compare` rule (#2910) * [docs] move pre-commit instructions to main dev docs (#2921) * [CI] Enable `testifylint` `expected-actual` rule (#2914) * [CI] Generate CRD json schema separately in pre-commit (#2930) * [release][1/N] Update YAMLs from Ray 2.9 to Ray 2.41 (#2934) * Delete `[raycluster|rayjob|rayservice]_types_test.go` unnecessary tests (#2935) * [release][2/N] Update RayCluster Helm chart from Ray 2.9 to Ray 2.41 (#2936) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [release][3/N] Update RayService e2e tests YAML files from Ray 2.9 to Ray 2.41 (#2937) * [release][4/N] Update Ray images / versions in kubectl plugin (#2938) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [release][5/N] Update some RayJob YAMLs from Ray 2.9 to Ray 2.41 (#2941) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [release][6/N] Remove unnecessary YAMLs (#2946) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [docs][kubectl-plugin] add dev docs (#2912) Signed-off-by: David Xia <david@davidxia.com> Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [CI] Add shellcheck and fix error of it (#2933) * [chore][kubectl-plugin] use better test assertions (#2955) Fix two places. * use `assert.EqualError()` instead of `assert.Error()` to check the error string is what we expect * use testify assertion instead of string checking for better error output Signed-off-by: David Xia <david@davidxia.com> * [chore] add Markdown linting pre-commit hook (#2953) Ignore most rules we violate for now. Fix these two violations. ``` pre-commit run markdownlint --all-files --show-diff-on-failure ... clients/python-client/README.md:106 MD001/heading-increment Heading levels should only increment by one level at a time [Expected: h3; Actual: h4] CHANGELOG.md:420:1 MD030/list-marker-space Spaces after list markers [Expected: 1; Actual: 2] ``` Signed-off-by: David Xia <david@davidxia.com> * [chore][kubectl-plugin] use consistent capitalization (#2950) in comments and console messages. capitalize "Ray" when used as a proper noun. Similar to #2922. Signed-off-by: David Xia <david@davidxia.com> * [Fix][RayJob] Invalid quote for RayJob submitter (#2949) Closes: #2943 Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [release][7/N] Update RayService YAMLs (#2956) * [CI] Enable testifylint len rule (#2945) Signed-off-by: LeoLiao123 <leoyeepaa@gmail.com> * [docs][kubectl-plugin] improve help messages (#2952) Signed-off-by: David Xia <david@davidxia.com> * [kubectl-plugin] Fix panic when GPU resource is not set (#2954) Signed-off-by: win5923 <ken89@kimo.com> * [release][8/N] Upgrade Stable Diffusion RayService to Ray 2.41 (#2960) * [docs][kubectl-plugin] fix incorrect example commands (#2951) Signed-off-by: David Xia <david@davidxia.com> Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [fix][kubectl-plugin] set worker group CPU limit (#2958) when creating a new worker group with `kubectl ray create workergroup`. Write a unit test. I noticed we are setting resource limits equal to resource requests everywhere else but in this command. I have a K8s [LimitRange] that prevented the creation of these worker Pods that had CPU limit defaulting to less than their CPU requests. Describing the RayCluster showed this warning event. `Failed to create worker Pod hyperkube/, Pod "dxia-test-other-group-worker-pm2sh" is invalid: spec.containers[0].resources.requests: Invalid value: "2": must be less than or equal to cpu limit of 250m` Signed-off-by: David Xia <david@davidxia.com> [LimitRange]: https://kubernetes.io/docs/concepts/policy/limit-range/ * [RayJob] Deflaky RayJob e2e tests (#2963) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [RayService] Deflaky RayService envtest (#2962) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [release][9/N] Update text summarizer RayService to Ray 2.41 (#2961) * [RayService] adapter vllm 0.6.1.post2 (#2823) * adapter vllm 0.6.1.post2 * fix var define * Unify the cpu Settings in serve.py in service.yaml, all set to 1 * Maintain the configuration with vllm0.5x * [Release] Upgrade ray-job.batch-inference.yaml image to 2.41 (#2971) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [chore][docs] enable Markdownlint rule MD010 (#2975) * [CI] Change Pre-commit-shellcheck-to-shellcheck-py (#2974) Signed-off-by: owenowenisme <mses010108@gmail.com> * [release] Update Yunikorn YAML file to Ray 2.41 (#2969) Signed-off-by: Cheng-Yeh Chung <kenchung285@gmail.com> * [release] Update YuniKorn YAML files to Ray 2.41 (#2976) Signed-off-by: win5923 <ken89@kimo.com> * [chore][docs] enable Markdownlint rule MD004 (#2973) [Unordered list style][1] [1]: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md004---unordered-list-style * [Test] Use GcsFaultToleranceOptions in test and backward compatibility (#2972) * Update samples to use Ray 2.41.0 images (#2964) Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * Update TPU Ray CR manifests to use Ray 2.41.0 (#2965) * [Refactor] Use constants for image tag, image repo, and versions in golang to avoid hard-coded strings (#2978) --------- Signed-off-by: Rueian <rueiancsie@gmail.com> Signed-off-by: kaihsun <kaihsun@anyscale.com> Signed-off-by: David Xia <david@davidxia.com> Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> Signed-off-by: 400Ping <43886578+400Ping@users.noreply.github.com> Signed-off-by: LeoLiao123 <leoyeepaa@gmail.com> Signed-off-by: win5923 <ken89@kimo.com> Signed-off-by: owenowenisme <mses010108@gmail.com> Signed-off-by: Cheng-Yeh Chung <kenchung285@gmail.com> Signed-off-by: Andrew Sy Kim <andrewsy@google.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Kai-Hsun Chen <kaihsun@anyscale.com> Co-authored-by: David Xia <david@davidxia.com> Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com> Co-authored-by: David Xia <dxia@spotify.com> Co-authored-by: Mykhailo Bobrovskyi <mikhail.bobrovsky@gmail.com> Co-authored-by: Ping <43886578+400Ping@users.noreply.github.com> Co-authored-by: Owen Lin <106612301+owenowenisme@users.noreply.github.com> Co-authored-by: Leo Liao <93932709+LeoLiao123@users.noreply.github.com> Co-authored-by: Blocka <ken89@kimo.com> Co-authored-by: zrant <37032227+pxp531@users.noreply.github.com> Co-authored-by: kenchung285 <kenchung285@gmail.com> Co-authored-by: fscnick <6858627+fscnick@users.noreply.github.com> Co-authored-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>

* [RayService] More envtests that follow the most common scenario in the RayService code path (ray-project#2880) Signed-off-by: Rueian <rueiancsie@gmail.com> * [RayService] Remove outdated env tests (ray-project#2886) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [RayService] Refactor envtests (ray-project#2888) * [docs][ray-operator] fix typo in Golang version (ray-project#2893) Project uses Golang 1.22 not 1.20. Signed-off-by: David Xia <david@davidxia.com> * [Fix][kubectl-plugin] Fix no context nil error SIGSEGV in tests (ray-project#2892) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [CI] Enable testifylint rule (ray-project#2896) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [CI] Enable `testifylint` `error-nil` rule (ray-project#2907) * [kubectl-plugin][feat] support specifying number of head GPUs (ray-project#2895) when creating a RayCluster with `kubectl ray create cluster NAME --head-gpu N`. Similar to the `--worker-gpu` switch. Signed-off-by: David Xia <david@davidxia.com> * Use webhook.CustomValidator instead of deprecated webhook.Validator. (ray-project#2803) * Use webhook.CustomValidator instead of deprecated webhook.Validator. * Move RayClusterWebhook to pkg/webhook/v1. * [kubectl-plugin] update context error messages (ray-project#2891) to tell user they can use `--context` to set the K8s context. Add tests Signed-off-by: David Xia <david@davidxia.com> Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [Fix][kubectl-plugin] make tests use a temporary kube config (ray-project#2894) file that's not at the default path. Right now tests fail if there's a K8s current context set with a command like `kubectl config use-context my-context`. This change allows tests to pass regardless of current context. Signed-off-by: David Xia <david@davidxia.com> * [CI] Enable testifylint formatter rule (ray-project#2915) Signed-off-by: 400Ping <43886578+400Ping@users.noreply.github.com> * [CI] Enable `testifylint` `empty` rule (ray-project#2908) * [CI] Enable testifylint bool-compare rule (ray-project#2911) Signed-off-by: 400Ping <43886578+400Ping@users.noreply.github.com> * [CI] Auto download golang tools in pre-commit (ray-project#2917) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [RayService] a safeguard for preventing overriding the pending cluster during a upgrade (ray-project#2887) Signed-off-by: Rueian <rueiancsie@gmail.com> * [RayService] Refactor unit tests for ShouldPrepareNewCluster (ray-project#2928) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [chore][kubectl-plugin] use consistent capitalization (ray-project#2922) in comments and console messages. * capitalize "Ray" when used as a proper noun * capitalize Ray K8s CRDs like "RayJob" and "RayCluster" * capitalize acronyms like "YAML" * fix some minor typos No functional changes. Signed-off-by: David Xia <david@davidxia.com> * [CI] Enable `testifylint` `require-error` rule (ray-project#2909) * [kubectl-plugin] support general `kubectl` switches like `--context` (ray-project#2883) * [CI] Fix lint error (require-error) (ray-project#2931) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [CI] Enable `testifylint` `float-compare` rule (ray-project#2910) * [docs] move pre-commit instructions to main dev docs (ray-project#2921) * [CI] Enable `testifylint` `expected-actual` rule (ray-project#2914) * [CI] Generate CRD json schema separately in pre-commit (ray-project#2930) * [release][1/N] Update YAMLs from Ray 2.9 to Ray 2.41 (ray-project#2934) * Delete `[raycluster|rayjob|rayservice]_types_test.go` unnecessary tests (ray-project#2935) * [release][2/N] Update RayCluster Helm chart from Ray 2.9 to Ray 2.41 (ray-project#2936) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [release][3/N] Update RayService e2e tests YAML files from Ray 2.9 to Ray 2.41 (ray-project#2937) * [release][4/N] Update Ray images / versions in kubectl plugin (ray-project#2938) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [release][5/N] Update some RayJob YAMLs from Ray 2.9 to Ray 2.41 (ray-project#2941) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [release][6/N] Remove unnecessary YAMLs (ray-project#2946) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [docs][kubectl-plugin] add dev docs (ray-project#2912) Signed-off-by: David Xia <david@davidxia.com> Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [CI] Add shellcheck and fix error of it (ray-project#2933) * [chore][kubectl-plugin] use better test assertions (ray-project#2955) Fix two places. * use `assert.EqualError()` instead of `assert.Error()` to check the error string is what we expect * use testify assertion instead of string checking for better error output Signed-off-by: David Xia <david@davidxia.com> * [chore] add Markdown linting pre-commit hook (ray-project#2953) Ignore most rules we violate for now. Fix these two violations. ``` pre-commit run markdownlint --all-files --show-diff-on-failure ... clients/python-client/README.md:106 MD001/heading-increment Heading levels should only increment by one level at a time [Expected: h3; Actual: h4] CHANGELOG.md:420:1 MD030/list-marker-space Spaces after list markers [Expected: 1; Actual: 2] ``` Signed-off-by: David Xia <david@davidxia.com> * [chore][kubectl-plugin] use consistent capitalization (ray-project#2950) in comments and console messages. capitalize "Ray" when used as a proper noun. Similar to ray-project#2922. Signed-off-by: David Xia <david@davidxia.com> * [Fix][RayJob] Invalid quote for RayJob submitter (ray-project#2949) Closes: ray-project#2943 Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [release][7/N] Update RayService YAMLs (ray-project#2956) * [CI] Enable testifylint len rule (ray-project#2945) Signed-off-by: LeoLiao123 <leoyeepaa@gmail.com> * [docs][kubectl-plugin] improve help messages (ray-project#2952) Signed-off-by: David Xia <david@davidxia.com> * [kubectl-plugin] Fix panic when GPU resource is not set (ray-project#2954) Signed-off-by: win5923 <ken89@kimo.com> * [release][8/N] Upgrade Stable Diffusion RayService to Ray 2.41 (ray-project#2960) * [docs][kubectl-plugin] fix incorrect example commands (ray-project#2951) Signed-off-by: David Xia <david@davidxia.com> Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [fix][kubectl-plugin] set worker group CPU limit (ray-project#2958) when creating a new worker group with `kubectl ray create workergroup`. Write a unit test. I noticed we are setting resource limits equal to resource requests everywhere else but in this command. I have a K8s [LimitRange] that prevented the creation of these worker Pods that had CPU limit defaulting to less than their CPU requests. Describing the RayCluster showed this warning event. `Failed to create worker Pod hyperkube/, Pod "dxia-test-other-group-worker-pm2sh" is invalid: spec.containers[0].resources.requests: Invalid value: "2": must be less than or equal to cpu limit of 250m` Signed-off-by: David Xia <david@davidxia.com> [LimitRange]: https://kubernetes.io/docs/concepts/policy/limit-range/ * [RayJob] Deflaky RayJob e2e tests (ray-project#2963) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [RayService] Deflaky RayService envtest (ray-project#2962) Signed-off-by: kaihsun <kaihsun@anyscale.com> * [release][9/N] Update text summarizer RayService to Ray 2.41 (ray-project#2961) * [RayService] adapter vllm 0.6.1.post2 (ray-project#2823) * adapter vllm 0.6.1.post2 * fix var define * Unify the cpu Settings in serve.py in service.yaml, all set to 1 * Maintain the configuration with vllm0.5x * [Release] Upgrade ray-job.batch-inference.yaml image to 2.41 (ray-project#2971) Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> * [chore][docs] enable Markdownlint rule MD010 (ray-project#2975) * [CI] Change Pre-commit-shellcheck-to-shellcheck-py (ray-project#2974) Signed-off-by: owenowenisme <mses010108@gmail.com> * [release] Update Yunikorn YAML file to Ray 2.41 (ray-project#2969) Signed-off-by: Cheng-Yeh Chung <kenchung285@gmail.com> * [release] Update YuniKorn YAML files to Ray 2.41 (ray-project#2976) Signed-off-by: win5923 <ken89@kimo.com> * [chore][docs] enable Markdownlint rule MD004 (ray-project#2973) [Unordered list style][1] [1]: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md004---unordered-list-style * [Test] Use GcsFaultToleranceOptions in test and backward compatibility (ray-project#2972) * Update samples to use Ray 2.41.0 images (ray-project#2964) Signed-off-by: Andrew Sy Kim <andrewsy@google.com> * Update TPU Ray CR manifests to use Ray 2.41.0 (ray-project#2965) * [Refactor] Use constants for image tag, image repo, and versions in golang to avoid hard-coded strings (ray-project#2978) --------- Signed-off-by: Rueian <rueiancsie@gmail.com> Signed-off-by: kaihsun <kaihsun@anyscale.com> Signed-off-by: David Xia <david@davidxia.com> Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com> Signed-off-by: 400Ping <43886578+400Ping@users.noreply.github.com> Signed-off-by: LeoLiao123 <leoyeepaa@gmail.com> Signed-off-by: win5923 <ken89@kimo.com> Signed-off-by: owenowenisme <mses010108@gmail.com> Signed-off-by: Cheng-Yeh Chung <kenchung285@gmail.com> Signed-off-by: Andrew Sy Kim <andrewsy@google.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Kai-Hsun Chen <kaihsun@anyscale.com> Co-authored-by: David Xia <david@davidxia.com> Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com> Co-authored-by: David Xia <dxia@spotify.com> Co-authored-by: Mykhailo Bobrovskyi <mikhail.bobrovsky@gmail.com> Co-authored-by: Ping <43886578+400Ping@users.noreply.github.com> Co-authored-by: Owen Lin <106612301+owenowenisme@users.noreply.github.com> Co-authored-by: Leo Liao <93932709+LeoLiao123@users.noreply.github.com> Co-authored-by: Blocka <ken89@kimo.com> Co-authored-by: zrant <37032227+pxp531@users.noreply.github.com> Co-authored-by: kenchung285 <kenchung285@gmail.com> Co-authored-by: fscnick <6858627+fscnick@users.noreply.github.com> Co-authored-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>

adapter vllm 0.6.1.post2

0438cd6

jannikstdl reviewed Feb 5, 2025

View reviewed changes

ray-operator/config/samples/vllm/serve.py Outdated Show resolved Hide resolved

ray-operator/config/samples/vllm/serve.py Outdated Show resolved Hide resolved

fix var define

218936b

andrewsykim reviewed Feb 6, 2025

View reviewed changes

ray-operator/config/samples/vllm/serve.py Outdated Show resolved Hide resolved

andrewsykim mentioned this pull request Feb 6, 2025

[release] Update the YAML files of the following examples to Ray 2.41. #2947

Closed

2 tasks

pxp531 added 2 commits February 7, 2025 09:43

Unify the cpu Settings in serve.py in service.yaml, all set to 1

bb9a7f9

Maintain the configuration with vllm0.5x

529ae7c

andrewsykim approved these changes Feb 7, 2025

View reviewed changes

andrewsykim merged commit f88b2fe into ray-project:master Feb 7, 2025
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[RayService] adapter vllm 0.6.1.post2 #2823

[RayService] adapter vllm 0.6.1.post2 #2823

Uh oh!

pxp531 commented Jan 26, 2025

Uh oh!

andrewsykim commented Feb 3, 2025

Uh oh!

jannikstdl commented Feb 3, 2025

Uh oh!

andrewsykim commented Feb 3, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

jannikstdl commented Feb 5, 2025

Uh oh!

andrewsykim commented Feb 6, 2025

Uh oh!

andrewsykim left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[RayService] adapter vllm 0.6.1.post2 #2823

[RayService] adapter vllm 0.6.1.post2 #2823

Uh oh!

Conversation

pxp531 commented Jan 26, 2025

Why are these changes needed?

Checks

Uh oh!

andrewsykim commented Feb 3, 2025

Uh oh!

jannikstdl commented Feb 3, 2025

Uh oh!

andrewsykim commented Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jannikstdl commented Feb 5, 2025

Uh oh!

andrewsykim commented Feb 6, 2025

Uh oh!

andrewsykim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewsykim commented Feb 3, 2025 •

edited

Loading