Skip to content

[kubectl-plugin] Add head/worker node selector option #3228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Mar 26, 2025
Merged

[kubectl-plugin] Add head/worker node selector option #3228

merged 2 commits into from
Mar 26, 2025

Conversation

troychiu
Copy link
Contributor

Changes

Add options for create cluster

  • --head-node-selectors
  • --worker-node-selectors

Why are these changes needed?

Node selectors are very common to set for RayCluster, especially since most providers use node selectors as a way to specify the GPU type.

Related issue number

Closes #3143

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(
❯ ./kubectl-ray create cluster test-cluster --head-node-selectors=1=2,3=4 --worker-node-selectors=a=b,c=d --dry-run
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: test-cluster
  namespace: default
spec:
  headGroupSpec:
    rayStartParams:
      dashboard-host: 0.0.0.0
    template:
      spec:
        containers:
        - image: rayproject/ray:2.41.0
          name: ray-head
          ports:
          - containerPort: 6379
            name: gcs-server
          - containerPort: 8265
            name: dashboard
          - containerPort: 10001
            name: client
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: "2"
              memory: 4Gi
        nodeSelector:
          "1": "2"
          "3": "4"
  rayVersion: 2.41.0
  workerGroupSpecs:
  - groupName: default-group
    rayStartParams:
      metrics-export-port: "8080"
    replicas: 1
    template:
      spec:
        containers:
        - image: rayproject/ray:2.41.0
          name: ray-worker
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: "2"
              memory: 4Gi
        nodeSelector:
          a: b
          c: d

troychiu added 2 commits March 25, 2025 23:38
@troychiu
add node selector option for kubectl plugin create cluster
79a9f70 
Signed-off-by: Troy Chiu <y.troychiu@gmail.com>
@troychiu
nit
b9195ce 
Signed-off-by: Troy Chiu <y.troychiu@gmail.com>
@MortalHappiness MortalHappiness mentioned this pull request Mar 26, 2025
Closed
4 tasks
@MortalHappiness MortalHappiness self-assigned this Mar 26, 2025
davidxia
davidxia approved these changes Mar 26, 2025
View reviewed changes
kubectl-plugin/pkg/cmd/create/create_cluster.go
Comment on lines +111 to +112
cmd.Flags().StringToStringVar(&options.headNodeSelectors, "head-node-selectors", nil, "Node selectors to apply to all head pods in the cluster (e.g. --head-node-selector=cloud.google.com/gke-accelerator=nvidia-l4,cloud.google.com/gke-nodepool=my-node-pool)")
cmd.Flags().StringToStringVar(&options.workerNodeSelectors, "worker-node-selectors", nil, "Node selectors to apply to all worker pods in the cluster (e.g. --worker-node-selector=cloud.google.com/gke-accelerator=nvidia-l4,cloud.google.com/gke-nodepool=my-node-pool)")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we separate with a space for readability and consistency with examples above?

Suggested change
cmd.Flags().StringToStringVar(&options.headNodeSelectors, "head-node-selectors", nil, "Node selectors to apply to all head pods in the cluster (e.g. --head-node-selector=cloud.google.com/gke-accelerator=nvidia-l4,cloud.google.com/gke-nodepool=my-node-pool)")
cmd.Flags().StringToStringVar(&options.workerNodeSelectors, "worker-node-selectors", nil, "Node selectors to apply to all worker pods in the cluster (e.g. --worker-node-selector=cloud.google.com/gke-accelerator=nvidia-l4,cloud.google.com/gke-nodepool=my-node-pool)")
cmd.Flags().StringToStringVar(&options.headNodeSelectors, "head-node-selectors", nil, "Node selectors to apply to all head pods in the cluster (e.g. --head-node-selector cloud.google.com/gke-accelerator=nvidia-l4,cloud.google.com/gke-nodepool=my-node-pool)")
cmd.Flags().StringToStringVar(&options.workerNodeSelectors, "worker-node-selectors", nil, "Node selectors to apply to all worker pods in the cluster (e.g. --worker-node-selector cloud.google.com/gke-accelerator=nvidia-l4,cloud.google.com/gke-nodepool=my-node-pool)")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@troychiu can you fix this in a follow-up as well as adding node selectors to the create workergroup command? https://github.com/ray-project/kuberay/blob/master/kubectl-plugin/pkg/cmd/create/create_workergroup.go

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes for sure! Thank you

@andrewsykim andrewsykim merged commit 13f7875 into ray-project:master Mar 26, 2025
21 checks passed
@troychiu troychiu mentioned this pull request Mar 27, 2025
Merged
4 tasks
andrewsykim pushed a commit to andrewsykim/kuberay that referenced this pull request Apr 2, 2025
@troychiu @andrewsykim
[kubectl-plugin] Add head/worker node selector option (ray-project#3228)
8dabcf4 
* add node selector option for kubectl plugin create cluster

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* nit

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

---------

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>
@andrewsykim andrewsykim mentioned this pull request Apr 2, 2025
Merged
4 tasks
andrewsykim pushed a commit to andrewsykim/kuberay that referenced this pull request Apr 2, 2025
@troychiu @andrewsykim
[kubectl-plugin] Add head/worker node selector option (ray-project#3228)
9f5f7f5 
* add node selector option for kubectl plugin create cluster

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* nit

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

---------

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>
andrewsykim pushed a commit to andrewsykim/kuberay that referenced this pull request Apr 2, 2025
@troychiu @andrewsykim
[kubectl-plugin] Add head/worker node selector option (ray-project#3228)
8f1587d 
* add node selector option for kubectl plugin create cluster

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* nit

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

---------

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>
andrewsykim added a commit that referenced this pull request Apr 2, 2025
@andrewsykim @rueian
@spencer-p @troychiu @tinaxfwu @MortalHappiness
[release-1.3] v1.3.2 backports (#3264)
4d53e84 
* [RayJob][Fix] Use --no-wait for job submission to avoid carrying the error return code to the log tailing (#3216)

* [RayJob][Fix] Use --no-wait for job submission to avoid carrying the error return code to the log tailing

Signed-off-by: Rueian <rueiancsie@gmail.com>

* [RayJob][Fix] Use --no-wait for job submission to avoid carrying the error return code to the log tailing

Signed-off-by: Rueian <rueiancsie@gmail.com>

* chore: update comments

Signed-off-by: Rueian <rueiancsie@gmail.com>

* chore: add a comment about bash -e

Signed-off-by: Rueian <rueiancsie@gmail.com>

---------

Signed-off-by: Rueian <rueiancsie@gmail.com>

* kubectl ray job submit: provide entrypoint (#3186)

* [kubectl-plugin] Add head/worker node selector option (#3228)

* add node selector option for kubectl plugin create cluster

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* nit

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

---------

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* add node selector option for kubectl plugin create worker group (#3235)

* add node selector option for kubectl plugin create work group

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* nit

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* code review: fix usage

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

---------

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* [kubectl-plugin] remove CPU limits by default (#3243)

Signed-off-by: Andrew Sy Kim <andrewsy@google.com>

* [Chore][CI] Limit the release-image-build github workflow to only take tag as input (#3117)

* remove all inputs from workflow_dispatch

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* use tag only

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* align case

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* change sha

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* extract tag

* lint fix

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* update github_env

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* directly take tag

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* add env,

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* directly use tag

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* use env. when in script

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* env.tag when with

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

* use env.tag for all

Signed-off-by: Tina Wu <j6vupz97@gmail.com>

---------

Signed-off-by: Tina Wu <j6vupz97@gmail.com>
Co-authored-by: tinaxfwu <twu@synchron.com>

* [CI] Remove create tag step from release (#3249)

Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>

---------

Signed-off-by: Rueian <rueiancsie@gmail.com>
Signed-off-by: Troy Chiu <y.troychiu@gmail.com>
Signed-off-by: Andrew Sy Kim <andrewsy@google.com>
Signed-off-by: Tina Wu <j6vupz97@gmail.com>
Signed-off-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
Co-authored-by: Rueian <rueiancsie@gmail.com>
Co-authored-by: Spencer Peterson <spencerjp@google.com>
Co-authored-by: Troy Chiu <114708546+troychiu@users.noreply.github.com>
Co-authored-by: Tina Wu <j6vupz97@gmail.com>
Co-authored-by: tinaxfwu <twu@synchron.com>
Co-authored-by: Chi-Sheng Liu <chishengliu@chishengliu.com>
win5923 pushed a commit to win5923/kuberay that referenced this pull request Apr 27, 2025
@troychiu @win5923
[kubectl-plugin] Add head/worker node selector option (ray-project#3228)
8366a36 
* add node selector option for kubectl plugin create cluster

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

* nit

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>

---------

Signed-off-by: Troy Chiu <y.troychiu@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewsykim andrewsykim andrewsykim approved these changes

@davidxia davidxia davidxia approved these changes

@MortalHappiness MortalHappiness MortalHappiness approved these changes

Assignees

@andrewsykim andrewsykim

@kevin85421 kevin85421

@MortalHappiness MortalHappiness

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feat][kubectl-plugin] Support node selectors for creating clusters
5 participants
@troychiu @davidxia @andrewsykim @MortalHappiness @kevin85421