Skip to content

[RayJob] add Light-weight RayJob Submitter #3943

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 44 commits into from
Sep 10, 2025
Merged

[RayJob] add Light-weight RayJob Submitter #3943

merged 44 commits into from
Sep 10, 2025

Conversation

owenowenisme
Copy link
Contributor

@owenowenisme owenowenisme commented Aug 13, 2025 •
edited
Loading

Why are these changes needed?

Closes #2537

This PR builds on @rueian’s work in PR #2587, with several improvements and refactoring. Many thanks to @rueian for the original contribution!

  1. Minimized Docker Image Size (from 54.4MB to 9.7MB, ~82% reduction):
  • Defined the ‎RayJobRequest type directly in the submitter code, avoiding the need to import the entire ray utils package. Note: If the type changes, updates will be required in two places.
  • Added ‎-ldflags="-w -s" to the build process to further optimize binary size.
❯ docker images
REPOSITORY                        TAG                         IMAGE ID       CREATED          SIZE
kuberay/submitter                 nightly-remove-import-ray   9e304545645e   4 seconds ago    9.7MB
kuberay/submitter                 nightly-import-ray          9b5000dc48cb   18 hours ago     54.4MB
  1. E2E Tests for Submitter
  • Added two end-to-end tests: one for a successful job and one for a failed job, using the lightweight submitter. These tests validate submitter functionality. I did not port all RayJob E2E tests, as those are specific to RayJob and do not impact the submitter.
  1. KubeRay CI & Image Release
  • Integrated E2E tests into the CI pipeline; all tests pass.
  • Further validation is needed for the image release workflow.
  1. Sample yaml for rayJob submitter
  • Created a sample yaml to indicate usage. Can be used with:
kubectl apply -f ray-operator/config/samples/ray-job.light-weight-submitter.yaml

Image Release Validation

To validate my changes in release image workflow, I ran the workflow on my local repo. And since I don't have quay account, I modified the image registry to ghcr on my local repo for testing purpose. (But this PR still uses quay)

I created a tag named v0.0.0-test-submitter-ci and manually run the build and release submitter image on it.
Workflow ran successfully and you can see that the image has been pushed to ghcr with multi-arch.
https://github.com/owenowenisme/kuberay/actions/runs/17117052455/job/48550113929

image

https://github.com/owenowenisme/kuberay/pkgs/container/submitter
image

image

Related issue number

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

@owenowenisme owenowenisme force-pushed the light-weight-submitter branch from 9806077 to c4e0317 Compare August 14, 2025 04:54
@owenowenisme
apply from rueian's poc with some modification
d8699e4 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
remove runtimeEnvYAML & entrypoint in spec of rayjob sample yaml
9306345 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
rename lw-submitter yaml metadata name
dbd5ce7 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
remove panic
591bae6 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
update
85fa4ae 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
add timeout to lw-submitter
ba022a0 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme owenowenisme force-pushed the light-weight-submitter branch from 903ad31 to ba022a0 Compare August 15, 2025 02:29
@owenowenisme
increase timeout
97e06f9 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
add e2e test
1f25222 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme owenowenisme force-pushed the light-weight-submitter branch from 6e62858 to 1f25222 Compare August 15, 2025 12:47
@owenowenisme
add pod log
18fe165 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
update build script
52f1575 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
remove debug log
071dd88 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
update e2e
5988005 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme owenowenisme marked this pull request as ready for review August 16, 2025 00:53
@owenowenisme
add multi-arch build and workflow release image
ed7a9b9 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme owenowenisme force-pushed the light-weight-submitter branch from 0f1a128 to ed7a9b9 Compare August 16, 2025 00:53
@owenowenisme
remove import of ray utils to minimize binary size
3a9e713 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
Copy link
Contributor Author

@Future-Outlier @rueian @kevin85421 PTAL🙏

fscnick
fscnick reviewed Aug 16, 2025
View reviewed changes
Copy link
Contributor

@fscnick fscnick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better to have an individual directory out of ray-operator? It seems the submitter's code doesn't rely on the ray-operator.

rueian
rueian reviewed Aug 16, 2025
View reviewed changes
@owenowenisme @fscnick
Update ray-operator/Dockerfile
a9bd1f8 
Co-authored-by: fscnick <6858627+fscnick@users.noreply.github.com>
Signed-off-by: Owen Lin (You-Cheng Lin) <106612301+owenowenisme@users.noreply.github.com>
@owenowenisme
Copy link
Contributor Author

owenowenisme commented Aug 17, 2025 •
edited
Loading

Would it be better to have an individual directory out of ray-operator? It seems the submitter's code doesn't rely on the ray-operator.

I have no strong opinion for this cc @rueian @Future-Outlier @kevin85421 WDYT?

Future-Outlier
Future-Outlier reviewed Aug 19, 2025
View reviewed changes
Future-Outlier
Future-Outlier reviewed Aug 19, 2025
View reviewed changes
Future-Outlier
Future-Outlier reviewed Aug 19, 2025
View reviewed changes
Future-Outlier
Future-Outlier reviewed Aug 19, 2025
View reviewed changes
Future-Outlier
Future-Outlier reviewed Aug 19, 2025
View reviewed changes
@owenowenisme
fix indent
1effdd7 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
modify existing submission id msg
b467c89 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
rename func
f2c2a8f 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
modify error msg
fb646a8 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
rename e2e test title name
05e6447 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
fix test
a4f8ec5 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme owenowenisme force-pushed the light-weight-submitter branch from e9eb9f5 to a4f8ec5 Compare September 8, 2025 13:06
@owenowenisme
fix test
69de1e7 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
fix test
bc4139c 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
fix test
4c88441 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
fix test
ee56d05 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
fix comment
3067da0 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
check submit job request status code
d6ffbdb 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme
check log in e2e
c3a61d3 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@Future-Outlier Future-Outlier mentioned this pull request Sep 9, 2025
Open
Future-Outlier
Future-Outlier reviewed Sep 9, 2025
View reviewed changes
Future-Outlier
Future-Outlier approved these changes Sep 9, 2025
View reviewed changes
Copy link
Member

@Future-Outlier Future-Outlier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, thank you for the hard work!

@owenowenisme owenowenisme force-pushed the light-weight-submitter branch 2 times, most recently from abdddc6 to 67ec8b5 Compare September 9, 2025 10:46
@owenowenisme
fix test
75eba73 
Signed-off-by: You-Cheng Lin (Owen) <mses010108@gmail.com>
@owenowenisme owenowenisme force-pushed the light-weight-submitter branch from 67ec8b5 to 75eba73 Compare September 9, 2025 12:21
@rueian
Update ray-operator/controllers/ray/utils/dashboardclient/dashboard_h…
ca77b5b 
…ttpclient.go

Signed-off-by: Rueian <rueiancsie@gmail.com>
owenowenisme
owenowenisme commented Sep 10, 2025
View reviewed changes
ray-operator/config/samples/ray-job.light-weight-submitter.yaml
restartPolicy: Never
containers:
- name: my-custom-rayjob-submitter
image: kuberay/submitter:nightly
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we'll need to change image here once the light-weight submitter image is released

@Future-Outlier
update
f7dad97 
Signed-off-by: Future-Outlier <eric901201@gmail.com>
@Future-Outlier Future-Outlier merged commit bd33d54 into ray-project:master Sep 10, 2025
24 of 27 checks passed
@ronaldo-valente-sgpiu
Copy link

@owenowenisme is this image published on hub.docker.com? or another CR?

@owenowenisme
Copy link
Contributor Author

owenowenisme commented Sep 10, 2025 •
edited
Loading

Hi, @ronaldo-valente-sgpiu, it will be released along with KubeRay 1.15.0 .
If you need the light-weight submitter now, you can build the image by running IMG=your-image-name make docker-image-rayjob-submitter in ray-operator directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewsykim andrewsykim andrewsykim left review comments

@kevin85421 kevin85421 kevin85421 left review comments

@rueian rueian rueian approved these changes

@Future-Outlier Future-Outlier Future-Outlier approved these changes

@MortalHappiness MortalHappiness Awaiting requested review from MortalHappiness MortalHappiness is a code owner

+1 more reviewer

@fscnick fscnick fscnick left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature] Light-weight job submitter
7 participants
@owenowenisme @rueian @fscnick @ronaldo-valente-sgpiu @andrewsykim @kevin85421 @Future-Outlier