[Feat][RayJob] UserMode SubmissionMode #2364

MortalHappiness · 2024-09-08T09:15:50Z

Why are these changes needed?

This PR implements the None SubmissionMode in this design doc. But after discussion we decided to rename it to UserMode.

Related issue number

N/A

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

MortalHappiness · 2024-09-08T09:19:44Z

ray-operator/controllers/ray/rayjob_controller_test.go

-				// Make sure the submission mode is K8sJobMode.
-				Expect(rayJob.Spec.SubmissionMode).To(Equal(rayv1.K8sJobMode))
+var _ = Context("RayJob with different submission modes", func() {
+	Context("RayJob in K8sJobMode", func() {


For reviewers: Please note that all the code in Context("RayJob in K8sJobMode") is simply indented. It may appear as though there are many changes, but in reality, nothing has actually changed.

MortalHappiness · 2024-09-08T09:21:08Z

ray-operator/controllers/ray/rayjob_controller_test.go

-		It("Make RayCluster.Status.State to be rayv1.Ready (attempt 2)", func() {
-			// The RayCluster is not 'Ready' yet because Pods are not running and ready.
-			Expect(rayCluster.Status.State).NotTo(Equal(rayv1.Ready)) //nolint:staticcheck // https://github.com/ray-project/kuberay/pull/2288
+	Context("RayJob in NoneMode", func() {


For reviewers: The real change is that I added Context("RayJob in NoneMode"), starting from here.

MortalHappiness · 2024-09-08T09:27:32Z

@kevin85421 PTAL

ray-operator/controllers/ray/rayjob_controller.go

kevin85421 · 2024-09-08T21:46:21Z

cc @andrewsykim

ray-operator/controllers/ray/rayjob_controller.go

andrewsykim · 2024-09-09T14:50:23Z

cc @chiayi for the CLI part

ray-operator/apis/ray/v1/rayjob_types.go

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

ray-operator/apis/ray/v1/rayjob_types.go

andrewsykim · 2024-09-12T14:38:17Z

Can you update the PR title?

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

kevin85421 · 2024-09-17T04:52:58Z

ray-operator/controllers/ray/rayjob_controller.go

 		if rayJobInstance.Spec.SubmissionMode == rayv1.K8sJobMode {
 			if err := r.createK8sJobIfNeed(ctx, rayJobInstance, rayClusterInstance); err != nil {
 				return ctrl.Result{RequeueAfter: RayJobDefaultRequeueDuration}, err
 			}
 		}

-		logger.Info("Both RayCluster and the submitter K8s Job are created. Transition the status from `Initializing` to `Running`.",
+		logger.Info("SubmissionMode is K8sJobMode and both RayCluster and the submitter K8s Job are created. Transition the status from `Initializing` to `Running`.",


Is it possible for HTTPMode going through here?

https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/rayjob-quick-start.html

kuberay/ray-operator/config/samples/ray-job.sample.yaml

Line 8 in d025792

# The alternative value is "HTTPMode", indicating that KubeRay will submit the Ray job by sending an HTTP request to the RayCluster.

You are right. I tested it by setting breakpoint in IDE. Log message updated.

ray-operator/controllers/ray/rayjob_controller.go

kevin85421 · 2024-09-17T05:02:03Z

ray-operator/controllers/ray/rayjob_controller.go

 			"RayJob", rayJobInstance.Name, "RayCluster", rayJobInstance.Status.RayClusterName)
 		rayJobInstance.Status.JobDeploymentStatus = rayv1.JobDeploymentStatusRunning
+	case rayv1.JobDeploymentStatusWaiting:
+		if rayJobInstance.Spec.SubmissionMode != rayv1.UserMode {


how about moving the check

kuberay/ray-operator/controllers/ray/rayjob_controller.go

Line 127 in d025792

if err := validateRayJobSpec(rayJobInstance); err != nil {

above?

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

danielgafni · 2024-10-06T20:05:05Z

Hey, is this feature considered available?

I can see it was included into KubeRay v1.2.2 release, but the docs haven't been updated to mention it.

MortalHappiness · 2024-10-07T01:32:49Z

@danielgafni This mode is only used in our kubectl plugin. Normal use cases should not directly use this.

danielgafni · 2024-10-09T10:25:59Z

What if I would like to have similar functionality with a different KubeRay interface (without using the kubectl plugin)?

Specifically, I'm thinking about local development against KubeRay with Dagster.

I'm working on dagster-ray integration and there are a few places where it would be possible to use the UserMode submission mode in order to quickly launch pipelines without building images.

As I understand, right now it should be possible to replicate what the plugin does:

Create a RayJob CR with submissionMode=UserMode
Use JobSubmissionClient with working_dir set in runtime_env to submit a job with some local files
Add the ray.io/ray-job-submission-id annotation to the RayJob CR

Are there any reasons this can't be done?

MortalHappiness · 2024-10-09T10:43:30Z

Sure. You can replicate what the kubectl plugin does manually.

@kevin85421 Do you think this use case is reasonable? Should we document this mode as a public API?

andrewsykim · 2024-10-09T13:04:39Z

I view the kubectl plugin as just a reference client implementation for the API. It was intended for users to create their own client / SDK with it as needed.

andrewsykim · 2024-10-09T13:06:00Z

Having said that, I think the API can use some improvements before we make it publicly available with documentation, mainly:

I think InteractiveMode is a better name than UserMode.
Instead of an annotation for the submission ID, add a new field spec.submissionId

MortalHappiness · 2024-10-09T13:35:35Z

I think adding a new field to spec is weird because spec is used to define the "desired state of the resource" but we don't know submission ID in advance. But anyway I agreed that this API needs more discussion and improvement before publicly document it.

andrewsykim · 2024-10-09T14:19:43Z

I think adding a new field to spec is weird because spec is used to define the "desired state of the resource" but we don't know submission ID in advance.

The field in spec is exactly for defining the "desired state", which is the pre-defined submission ID that a user has to use for job submission. It would simplify Daniel's step from #2364 (comment) to the following:

Create a RayJob CR with submissionMode=UserMode and spec.submissionId="some-id"
Use JobSubmissionClient with working_dir set in runtime_env to submit a job with some local files, using submission ID from 1)

I also think that something like spec.submissionId can be used for other submission modes to "hard code" the job to use a well-defined submission ID that is easier to search in logs

andrewsykim · 2024-10-09T14:21:14Z

We already released 1.2.2 with the API so I'd be worried about changing it at this point and breaking people, see #2418

kevin85421 · 2024-10-09T18:25:12Z

Hi @danielgafni, we will make it a public API in the next release. However, we may introduce some breaking changes. If it's not urgent, you can wait for the next release.

Do you think this use case is reasonable? Should we document this mode as a public API?

@MortalHappiness the use case makes sense to me, but I think we should wait until the API is more stable before documenting it.

I'd be worried about changing it at this point and breaking people

I think it is fine. It is an alpha API, and currently only kubectl plugin should use it.

danielgafni · 2024-10-09T21:06:56Z

Thanks, I'm fine with trying this feature in experimental setting.

danielgafni · 2024-10-14T09:30:27Z

The field in spec is exactly for defining the "desired state", which is the pre-defined submission ID

@andrewsykim Will this approach work for running interactive ray jobs in client mode? I would like the RayJob CR to manage the cluster lifecycle but also to be able to connect in client mode.

I don't know if a predefined job_id can be used with ray.init.

The other way would definitely work tho - it's trivial to first run ray.init and then add an annotation with the generated job id.

andrewsykim · 2024-10-16T14:01:05Z

@danielgafni you can submit a job and specfiy --submission-id with ray job submit where the python script calls ray.init(). But RayJob itself is not really meant to be used with interactive client like you would with RayCluster

danielgafni · 2024-10-16T19:12:45Z

you can submit a job and specfiy --submission-id with ray job submit where the python script calls ray.init()

Not really what I'm looking for, I was hoping it would be possible to just call ray.init from Python code without breaking out of the main code flow.

RayJob itself is not really meant to be used with interactive client

Right now it isn't but don't you think the possibility of changing that would be quite intriguing?

It seems like this workflow should be very possible (with the annotation approach).

andrewsykim · 2024-10-16T19:17:57Z

If you can specify and override the submission ID for an interactive session with ray.init(), then I think it's possible. I don't recall if that's possible.

danielgafni · 2024-10-17T11:38:57Z

I don't think so. Perhaps introducing this option to ray.init would be another path towards the interactive RayJob workflow.

danielgafni · 2025-08-11T12:15:10Z

Hey guys, is this functionality considered public & stable at this point? I want to implement the following in dagster-ray:

Create a RayJob CR with submissionMode=InteractiveMode

Use JobSubmissionClient to submit a job with some local files. Alternatively, use ray.init.

Add the ray.io/ray-job-submission-id annotation to the RayJob CR

MortalHappiness commented Sep 8, 2024

View reviewed changes

MortalHappiness force-pushed the feat/rayjob-none-submission-mode branch 3 times, most recently from e36ec2d to c0e1e56 Compare September 8, 2024 11:50

kevin85421 reviewed Sep 8, 2024

View reviewed changes

ray-operator/controllers/ray/rayjob_controller.go Outdated Show resolved Hide resolved

ray-operator/controllers/ray/rayjob_controller.go Outdated Show resolved Hide resolved

ray-operator/controllers/ray/rayjob_controller.go Show resolved Hide resolved

kevin85421 self-assigned this Sep 8, 2024

andrewsykim reviewed Sep 9, 2024

View reviewed changes

ray-operator/controllers/ray/rayjob_controller.go Show resolved Hide resolved

MortalHappiness marked this pull request as draft September 9, 2024 15:04

MortalHappiness force-pushed the feat/rayjob-none-submission-mode branch from 59cd4f6 to a111d85 Compare September 9, 2024 15:06

MortalHappiness marked this pull request as ready for review September 9, 2024 15:09

MortalHappiness force-pushed the feat/rayjob-none-submission-mode branch from a111d85 to 3288ae7 Compare September 10, 2024 12:30

andrewsykim reviewed Sep 10, 2024

View reviewed changes

ray-operator/apis/ray/v1/rayjob_types.go Outdated Show resolved Hide resolved

kevin85421 added the 1.3.0 label Sep 10, 2024

MortalHappiness force-pushed the feat/rayjob-none-submission-mode branch 2 times, most recently from 35197ba to d0a1dab Compare September 11, 2024 11:58

andrewsykim mentioned this pull request Sep 12, 2024

[Question] Clarify how to use JobSubmissionClient with jobs created by RayJob #2348

Closed

1 task

MortalHappiness added 6 commits September 12, 2024 16:45

[Feat][RayJob] Add new JobDeploymentStatus and JobSubmissionMode

42a0d7e

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

[Feat][RayJob] Mark entrypoint as optional

86dc8e3

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

[Feat][RayJob] Finish RayJob NoneMode SubmissionMode

f6ed64a

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

[Test][RayJob] Add tests for NodeMode submission mode

b8ca7e9

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

[Chore][RayJob] Update according to PR comments

54b5a08

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

[Chore][RayJob] Rename NoneMode to UserMode

d0be059

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

MortalHappiness force-pushed the feat/rayjob-none-submission-mode branch from d0a1dab to d0be059 Compare September 12, 2024 08:45

andrewsykim reviewed Sep 12, 2024

View reviewed changes

ray-operator/apis/ray/v1/rayjob_types.go Outdated Show resolved Hide resolved

MortalHappiness changed the title ~~[Feat][RayJob] NoneMode SubmissionMode~~ [Feat][RayJob] UserMode SubmissionMode Sep 12, 2024

[Chore][RayJob] Update comments for UserMode

b4ff999

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

MortalHappiness force-pushed the feat/rayjob-none-submission-mode branch from 12351a2 to b4ff999 Compare September 12, 2024 14:45

kevin85421 reviewed Sep 17, 2024

View reviewed changes

MortalHappiness added 2 commits September 17, 2024 19:03

[Chore][RayJob] Update log messages

38f92b3

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

[Refactor][RayJob] Extract validateRayJobStatus as a function

04729ba

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

MortalHappiness force-pushed the feat/rayjob-none-submission-mode branch from 1bb275e to 04729ba Compare September 17, 2024 11:12

kevin85421 approved these changes Sep 17, 2024

View reviewed changes

kevin85421 merged commit cf4a877 into ray-project:master Sep 17, 2024
27 checks passed

chiayi mentioned this pull request Sep 21, 2024

[Feat][Kubectl-Plugin]Implement kubectl ray job submit #2394

Merged

4 tasks

danielgafni mentioned this pull request Oct 9, 2024

Support UserMode RayJob submission mode for local testing danielgafni/dagster-ray#31

Open

andrewsykim mentioned this pull request Oct 16, 2024

[RayJob] UserMode -> InteractiveMode and check rayjob.spec.jobId instead of annotation #2446

Merged

4 tasks

[Feat][RayJob] UserMode SubmissionMode #2364

[Feat][RayJob] UserMode SubmissionMode #2364

Uh oh!

Conversation

MortalHappiness commented Sep 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

MortalHappiness Sep 8, 2024

Choose a reason for hiding this comment

Uh oh!

MortalHappiness Sep 8, 2024

Choose a reason for hiding this comment

Uh oh!

MortalHappiness commented Sep 8, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevin85421 commented Sep 8, 2024

Uh oh!

Uh oh!

andrewsykim commented Sep 9, 2024

Uh oh!

Uh oh!

Uh oh!

andrewsykim commented Sep 12, 2024

Uh oh!

kevin85421 Sep 17, 2024

Choose a reason for hiding this comment

Uh oh!

kevin85421 Sep 17, 2024

Choose a reason for hiding this comment

Uh oh!

kevin85421 Sep 17, 2024

Choose a reason for hiding this comment

Uh oh!

MortalHappiness Sep 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kevin85421 Sep 17, 2024

Choose a reason for hiding this comment

Uh oh!

MortalHappiness Sep 17, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danielgafni commented Oct 6, 2024

Uh oh!

MortalHappiness commented Oct 7, 2024

Uh oh!

danielgafni commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MortalHappiness commented Oct 9, 2024

Uh oh!

andrewsykim commented Oct 9, 2024

Uh oh!

andrewsykim commented Oct 9, 2024

Uh oh!

MortalHappiness commented Oct 9, 2024

Uh oh!

andrewsykim commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewsykim commented Oct 9, 2024

Uh oh!

kevin85421 commented Oct 9, 2024

Uh oh!

danielgafni commented Oct 9, 2024

Uh oh!

danielgafni commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewsykim commented Oct 16, 2024

MortalHappiness commented Sep 8, 2024 •

edited

Loading

MortalHappiness Sep 17, 2024 •

edited

Loading

danielgafni commented Oct 9, 2024 •

edited

Loading

andrewsykim commented Oct 9, 2024 •

edited

Loading

danielgafni commented Oct 14, 2024 •

edited

Loading