Add label selector scheduling logic to Cluster Resource Manager #51901

ryanaoleary · 2025-04-02T04:53:00Z

Why are these changes needed?

This PR updates the cluster resource scheduling logic to check whether an eligible node satisfies the given label match expressions when checking if a node IsSchedulable. This PR also adds the label_selector option to Task/Actor creation, adds logic to parse strings to the LabelSelector data structure in the raylet, and passes the label_selector to the core worker to be used when building the TaskSpec.

These changes are to support the label selector API to ensure tasks/actors execute on nodes with the required node labels.

Related issue number

#51564

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Fix types Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Fix proto Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Update proto naming to match autoscaler Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Fix errors Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Remove gcs proto Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Fix header file Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Change expression to constraint Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Format Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Add label_selector to common task spec Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Signed-off-by: ryanaoleary <ryanaoleary@google.com>

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

… label_selector API Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

python/ray/_raylet.pyx

…pare_label_selector` (#52964) This PR is a follow-up to this comment: #51901 (comment). This PR changes the cluster resource scheduler to propagate a Ray status to `ComputeResources` in `TaskSpecification` when the LabelSelector data type is initialized. This allows a task built with a malformed label selector to return an error as a more useful Python exception rather than crashing Ray components in the C++. #51564 --------- Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

…pare_label_selector` (#52964) This PR is a follow-up to this comment: #51901 (comment). This PR changes the cluster resource scheduler to propagate a Ray status to `ComputeResources` in `TaskSpecification` when the LabelSelector data type is initialized. This allows a task built with a malformed label selector to return an error as a more useful Python exception rather than crashing Ray components in the C++. #51564 --------- Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Kamil Kaczmarek <kamil@anyscale.com>

…pare_label_selector` (ray-project#52964) This PR is a follow-up to this comment: ray-project#51901 (comment). This PR changes the cluster resource scheduler to propagate a Ray status to `ComputeResources` in `TaskSpecification` when the LabelSelector data type is initialized. This allows a task built with a malformed label selector to return an error as a more useful Python exception rather than crashing Ray components in the C++. ray-project#51564 --------- Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Michael Acar <michael.j.acar@gmail.com>

…pare_label_selector` (#52964) This PR is a follow-up to this comment: #51901 (comment). This PR changes the cluster resource scheduler to propagate a Ray status to `ComputeResources` in `TaskSpecification` when the LabelSelector data type is initialized. This allows a task built with a malformed label selector to return an error as a more useful Python exception rather than crashing Ray components in the C++. #51564 --------- Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…pare_label_selector` (#52964) This PR is a follow-up to this comment: #51901 (comment). This PR changes the cluster resource scheduler to propagate a Ray status to `ComputeResources` in `TaskSpecification` when the LabelSelector data type is initialized. This allows a task built with a malformed label selector to return an error as a more useful Python exception rather than crashing Ray components in the C++. #51564 --------- Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: sampan <sampan@anyscale.com>

ryanaoleary force-pushed the schedule-using-labels branch from 87451d8 to 2e93936 Compare April 2, 2025 10:37

ryanaoleary added 2 commits April 2, 2025 10:41

Remove unused var

cc66813

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Format

41405e1

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

ryanaoleary mentioned this pull request Apr 2, 2025

[Core] Ray Label Selector API Implementation Tracker #51564

Open

23 tasks

ryanaoleary added 2 commits April 2, 2025 12:12

Add missing const keyword

47e24da

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Clang format

c079fac

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

jcotant1 added the core Issues that should be addressed in Ray Core label Apr 2, 2025

ryanaoleary mentioned this pull request Apr 3, 2025

Add label_selector option to remote functions #51707

Merged

8 tasks

hainesmichaelc added the community-contribution Contributed by the community label Apr 4, 2025

ryanaoleary and others added 16 commits April 8, 2025 06:31

Add parse label_selector logic

97c5341

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Remove actor logic from this PR

1c9f7f0

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Add label_selector to common task spec

a8f0d4c

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Add label_selector to common task spec Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Add label_selector to common task and actor options

7f473d2

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Pass actor to core worker

2cbe222

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Fix errors

3602496

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Fix task util

21d48f0

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Format

a002e48

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Merge branch 'master' into schedule-using-labels

51d6c9a

Add label_selector to TaskSpec proto

17961a0

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Fix types

2926164

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Fix cython build errors

4e2108c

Signed-off-by: ryanaoleary <ryanaoleary@google.com>

Merge branch 'master' into schedule-using-labels

6deb424

Merge branch 'master' into schedule-using-labels

754a69b

Fix task spec args and add in remote function PR changes

9144d87

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Fix task spec args and add in remote function PR changes

4af1476

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

ryanaoleary marked this pull request as ready for review April 15, 2025 13:18

ryanaoleary requested review from pcmoritz, raulchen and a team as code owners April 15, 2025 13:18

ryanaoleary and others added 8 commits May 1, 2025 21:38

Remove throw on empty value - since this is actually allowed with the…

d1892aa

… label_selector API Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Update src/ray/common/scheduling/label_selector.h

d0a16d0

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/common/scheduling/label_selector.h

d7f8b94

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/common/scheduling/label_selector.h

7a79833

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/common/scheduling/label_selector.cc

6f603a1

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/common/scheduling/label_selector.cc

ce9547c

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Fix nits and error logging

8b4b0d4

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Merge branch 'master' into schedule-using-labels

7b0f958

ryanaoleary requested a review from dayshah May 2, 2025 02:12

ryanaoleary and others added 4 commits May 3, 2025 00:22

Fix failing test

108722a

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Merge branch 'master' into schedule-using-labels

f619c9d

Fix broken label test

14112c2

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Merge branch 'master' into schedule-using-labels

de71cf2

edoakes reviewed May 6, 2025

View reviewed changes

python/ray/_raylet.pyx Show resolved Hide resolved

edoakes approved these changes May 6, 2025

View reviewed changes

edoakes merged commit f6e9ca2 into ray-project:master May 6, 2025
5 checks passed

ryanaoleary mentioned this pull request May 13, 2025

[Core] Remove RAY_LOG(ERROR from LabelSelector and validate in prepare_label_selector #52964

Merged

8 tasks

hainesmichaelc added the community-backlog label May 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add label selector scheduling logic to Cluster Resource Manager #51901

Add label selector scheduling logic to Cluster Resource Manager #51901

Uh oh!

ryanaoleary commented Apr 2, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add label selector scheduling logic to Cluster Resource Manager #51901

Add label selector scheduling logic to Cluster Resource Manager #51901

Uh oh!

Conversation

ryanaoleary commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanaoleary commented Apr 2, 2025 •

edited

Loading