[Core] Update V2 Autoscaler to support scheduling using Node labels and LabelSelector API #53578

ryanaoleary · 2025-06-05T10:38:01Z

Why are these changes needed?

This PR updates the GCS state manager to append the labels set on a Node and retrieved from GCS to the dynamic_node_labels on the NodeState passed to the autoscaler. This PR also updates the V2 autoscaler to consider labels when calling _compute_score to schedule a ResourceRequest on a Node, assigning a higher score to a node type with labels that satisfy a LabelSelector.

Related issue number

#51564

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

…pute_score Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

ryanaoleary · 2025-06-05T10:39:17Z

@MengjinYan I ended up implementing the logic to pass labels from GCS to the autoscaler and consider labels in the autoscaler when scheduling in the same PR because it made it easier to test the change.

can-anyscale

some tests are failing, could be related @ryanaoleary

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

MengjinYan

Looks like we should also add the logic to parse the LabelSelector from the GCS GetClusterStatusReply in autoscaler to populate the label selector information.

Also, is it possible to add end to end test with label selector?

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

python/ray/autoscaler/v2/tests/test_scheduler.py

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

src/ray/gcs/gcs_server/state_util.cc

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

MengjinYan · 2025-07-02T21:46:58Z

@rueian Can you take a look at the PR as well from autoscaler's perspective to see if we missed anything? Thanks!

python/ray/autoscaler/v2/scheduler.py

MengjinYan · 2025-07-03T06:26:22Z

python/ray/autoscaler/v2/scheduler.py

Nit: I think we should also change the _sort_resource_request logic in _try_schedule to add labels to the sorting mechanism, as there can resource requests with exact same resource requirements but different label selectors

Sounds good to me, done in 8cb1816. I had it sort by the length of the constraints of the first selector, similar to how we sort based on the length of the placement constraints.

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

rueian · 2025-07-08T17:56:14Z

src/ray/protobuf/gcs.proto

@@ -493,6 +493,8 @@ message ResourceDemand {
  // The number of requests of this shape still queued in CoreWorkers that this
  // raylet knows about.
  int64 backlog_size = 4;
+  // The label selector constraints for this Resource shape on a node.
+  repeated LabelSelector label_selectors = 5;


There is already a repeated LabelSelectorConstraint field in the LabelSelector. Why do we need repeated here?

The LabelSelectorConstraints are the conditions associated with one label selector with AND semantics (i.e. we require a node to have labels accelerator-type="TPU" and market-type="SPOT" if those two respective constraints are specified). The repeated label selector is so that we can support fallback with label selectors (i.e. the user specifies two separate label selectors with their own constraint(s), and we fallback to the second one if the first is unsatisfiable).

src/ray/gcs/gcs_server/gcs_resource_manager.cc

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

dayshah

generally just minor nits, not really blockers

src/ray/common/scheduling/label_selector.cc

src/ray/common/scheduling/label_selector.h

src/ray/gcs/gcs_server/test/gcs_autoscaler_state_manager_test.cc

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

src/ray/gcs/gcs_server/state_util.cc

src/ray/gcs/gcs_server/state_util.h

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.h

src/ray/gcs/gcs_server/state_util.h

dayshah · 2025-07-28T21:04:54Z

src/ray/common/scheduling/label_selector.h

+
+namespace std {
+template <>
+struct hash<ray::LabelSelector> {


are we using std sets/maps with this somewhere instead of absl ones. Would prefer sticking to absl and just defining the absl hash functions

We are using std::hash in task_spec.h here for the scheduling class descriptor. I think since we're using absl::flat_hash_map in that class I can just go ahead and switch it all to use absl hash functions instead.

Done in 9b3bbbe.

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

…nd LabelSelector API (ray-project#53578) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Krishna Kalyan <krishnakalyan3@gmail.com>

…nd LabelSelector API (ray-project#53578) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: avigyabb <avigyabb@stanford.edu>

…nd LabelSelector API (#53578) Signed-off-by: Ryan O'Leary <ryanaoleary@google.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com> Co-authored-by: Rueian <rueiancsie@gmail.com> Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Kamil Kaczmarek <kamil@anyscale.com>

ryanaoleary added 2 commits June 5, 2025 10:22

Pass node labels from gcs to autoscaler and use label_selector in com…

1233494

…pute_score Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Update test

94b61fb

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

ryanaoleary requested a review from a team as a code owner June 5, 2025 10:38

ryanaoleary mentioned this pull request Jun 5, 2025

[Core] Ray Label Selector API Implementation Tracker #51564

Open

23 tasks

can-anyscale reviewed Jun 10, 2025

View reviewed changes

ryanaoleary and others added 2 commits June 13, 2025 12:46

Fix pg_usage_labels test

247af51

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Merge branch 'master' into autoscaler-labels

e603c00

MengjinYan reviewed Jun 13, 2025

View reviewed changes

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc Outdated Show resolved Hide resolved

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc Show resolved Hide resolved

python/ray/autoscaler/v2/tests/test_scheduler.py Show resolved Hide resolved

ryanaoleary added 2 commits June 24, 2025 01:03

Propogate labelselector info

f991648

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Pass label selector info and add/update tests

b51d643

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

ryanaoleary requested review from pcmoritz and raulchen as code owners June 25, 2025 08:10

ryanaoleary and others added 5 commits June 25, 2025 08:27

Merge branch 'master' into autoscaler-labels

161a8a4

Update c++ test to check all operators

9d00796

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

clang format

f6528bb

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Merge branch 'master' into autoscaler-labels

8adaa0e

Fix c++ test

7d593ea

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

MengjinYan reviewed Jul 1, 2025

View reviewed changes

MengjinYan requested a review from rueian July 2, 2025 21:47

MengjinYan reviewed Jul 3, 2025

View reviewed changes

Move LabelSelector message to common proto and add scheduling descriptor

bb9420f

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

rueian reviewed Jul 8, 2025

View reviewed changes

src/ray/gcs/gcs_server/gcs_resource_manager.cc Show resolved Hide resolved

rueian reviewed Jul 8, 2025

View reviewed changes

src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc Show resolved Hide resolved

ryanaoleary and others added 4 commits July 9, 2025 03:28

Fix scheduling logic and add e2e test

8cb1816

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Fix remaining comments

18aa109

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Merge branch 'master' into autoscaler-labels

ece9c9d

Fix FillAggregateLoad function

55c94f0

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

rueian approved these changes Jul 23, 2025

View reviewed changes

edoakes approved these changes Jul 23, 2025

View reviewed changes

edoakes added the go add ONLY when ready to merge, run all tests label Jul 23, 2025

Merge branch 'master' into autoscaler-labels

c0a3f2c

ray-gardener bot added community-contribution Contributed by the community core Issues that should be addressed in Ray Core labels Jul 23, 2025

ryanaoleary added 2 commits July 24, 2025 00:43

Fix lint

c1a0eb4

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Undo pydoclint change

0e8bf9f

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

dayshah reviewed Jul 28, 2025

View reviewed changes

ryanaoleary and others added 10 commits July 28, 2025 21:44

Update src/ray/common/scheduling/label_selector.h

20d0737

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

f799d11

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

49e1ef4

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.h

8a06698

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

65266a2

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc

d0270ca

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Update src/ray/gcs/gcs_server/gcs_resource_manager.cc

7a04a3e

Co-authored-by: Dhyey Shah <dhyey2019@gmail.com> Signed-off-by: Ryan O'Leary <113500783+ryanaoleary@users.noreply.github.com>

Remove friend tests, use absl hash, fix nit comments

9b3bbbe

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Use move() and reserve vector length

1af13a1

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

remove erroneous duplicate for loop

701b97a

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

ryanaoleary requested a review from dayshah July 29, 2025 09:21

ryanaoleary and others added 4 commits July 29, 2025 09:56

Remove pydoclint baseline change

c1a302c

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

Merge branch 'master' into autoscaler-labels

c8bc79a

Merge branch 'master' into autoscaler-labels

a4ee6c4

Fix pdyoclint-baseline

4741c40

Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>

jjyao changed the title ~~Update V2 Autoscaler to support scheduling using Node labels and LabelSelector API~~ [Core] Update V2 Autoscaler to support scheduling using Node labels and LabelSelector API Jul 30, 2025

jjyao merged commit ffffcd4 into ray-project:master Jul 30, 2025
5 checks passed

[Core] Update V2 Autoscaler to support scheduling using Node labels and LabelSelector API #53578

[Core] Update V2 Autoscaler to support scheduling using Node labels and LabelSelector API #53578

Uh oh!

Conversation

ryanaoleary commented Jun 5, 2025

Why are these changes needed?

Related issue number

Checks

Uh oh!

ryanaoleary commented Jun 5, 2025

Uh oh!

can-anyscale left a comment

Choose a reason for hiding this comment

Uh oh!

MengjinYan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MengjinYan commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

MengjinYan Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

ryanaoleary Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

rueian Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

ryanaoleary Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dayshah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dayshah Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

ryanaoleary Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

ryanaoleary Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MengjinYan left a comment •

edited

Loading