-
Notifications
You must be signed in to change notification settings - Fork 29
Ray Label Based Scheduling REP #60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great @ryanaoleary!
Could you please add sections detailing:
- Requirements for label keys and values (basically same as k8s)
- The syntax for label selector operators
- The set of default labels that will be supported initially
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com> Signed-off-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>
Co-authored-by: Mengjin Yan <mengjinyan3@gmail.com> Signed-off-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few minor comments, then LGTM to broadcast
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: ryanaoleary <113500783+ryanaoleary@users.noreply.github.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
|
||
We will also support sourcing labels from a file using bash for `ray start` only. This command will read labels in YAML format to support passing down Pod labels into the Raylet using downward API. The labels passed in from file should be composable with those specified by `--labels`, with the value in `--labels` taking precedence if there is a conflict. | ||
```sh | ||
ray start --labels-file $(cat labels.txt) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be:
ray start --labels-file labels.txt
Or more specifically it should be an absolute path
ray start --labels-file /path/to/labels.yaml
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I started implementing it here: https://github.com/ray-project/ray/pull/51706/files as --labels-from-file
(I need to update the name) where it just sources the contents of the file with $(cat labels.txt)
and expects a string. If we want to read directly from a file instead I can change it. Should we support both txt and yaml files (I guess both would be expected in yaml format)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we just want to source from a file with cat labels.txt
we don't need to add anything inside of Ray -- bash does the trick :)
Andrew mentioned it'd be better to support reading YAML files for the "downward" API, that's the impetus for supporting --labels-file
explicitly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah agreed, $(cat labels.txt)
is perfectly fine, but you would pass that into ray start --labels
, not --labels-file
was my point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So if I understand correctly, for the string input, no matter if it is specified in the command or if it is sourcing from a file, we want to use the --labels
. And in both cases, we expects the input format to be key1=value1,key2=value2
.
And at the same time, if we want to support using "downward" API with yamal file as well, the idea is either to have a --labels-file
option and pass in the yaml file name so that ray start
can read the file directory, or more simply, just use the --labels
options and add the file read & translation logic outside of the ray logic.
And I believe we can always start from the simpler solution. Is that right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh I see, I was expecting --labels-file
to source from the file but just to expect the string in yaml format (so both options would take a string but then parse them from different formats). I updated the REP to reflect reading directly from a file in 3c444e7. I'll update ray-project/ray#51706 to that implementation as well.
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
This REP is a continuation of the node affinity feature enhancement REP and based on the public proposal for a Ray Label Selector API.