are there any other runtime_context fields that require RPCs to the GCS?

let's at least lazily initialize & then cache this value, so if users call it in a tight loop it won't require an RPC on every invocation (labels are static for now)

+1

Done in 62ff362. I don't see other runtime context fields calling to GCS, stuff like get_current_node_id call to the core worker, but I don't think we can get the node labels from there without calling to GCS/the node table. An alternative to calling services.get_node in the worker would be adding a field self._node_labels in the Node class and setting it here when it's initialized from node_info. I think the issue is that on connect_only calls to the Node, the RayParams are empty and so the value returned is None.

Hm it seems like this existing codepath relies on the ray params in the connect_only branch:

Getting it from the Node class is more in line with how the rest of the runtime context API is implemented, so if we can do that in a way that's correct I'd prefer it. I'll leave it to you to briefly investigate if this works in all cases, and if not we can go with the current state of the PR.

Yeah I agree in the Node class is better, since I think it'd be possible for a worker process to connect to a different node right (and with the cache implementation we'd only ever return the first node's labels). I didn't see at that point in the connect_only path that it uses the RayParams unless _plasma_store_socket_name is None. I still think it makes more sense to query the node_info here rather than in the worker, since node __init__ already makes these calls in both branches in the above location and here to set vars. In
2392754 I updated the logic to call get_node if connect_only and retrieve the node labels if set.

let's also add a condition that checks the labels from runtime context from the driver

Sorry just saw this comment, added a check in 516cc14 that the node labels from the driver runtime context should be empty