Current problem

Currently the uv ray integration works out of the box, but leads to very slow startup. When we enabled it, jobs took on the order of 1-5 minutes to start, and the isolated environments took a while to build despite setting UV_CACHE_DIR. Some users even reported hangs during the dependency building of the workers.

Short-term Workaround

The workaround for v0.1.0 was to not use the isolation provided by the uv-ray integration, but rather just set the py_executable to the virtual environment that the driver sets up (see DEFAULT_VENV). This lead to very fast start times, but at the cost of no isolation. This works because all our dependencies do not conflict, but in the near future, this will not hold true as we add more libraries

Other slowdowns

We also observed that when the py_executable is set to uv run, it can be made faster by setting --cache-dir on top of setting UV_CACHE_DIR when launching the driver script (see this). It's not clear why this helps, but it seems like the command for py_executable may not take into account the cache-dir the user specifies in their environment.

Feature request

We want to be able to use the other PY_EXECUTABLES (or arbitrary ones) and not incur this huge build time. This issue tracks the overall dependency isolation via uv. Isolation can also be accomplished via containers, but that is beyond the scope of this issue.