fix: metaflow-dev improvements #2527
Conversation
| # We read the actual location from package metadata in this case, but only do this heavier operation if the above lookups fail. | ||
| try: | ||
| import json | ||
| from importlib.metadata import Distribution |
There was a problem hiding this comment.
for py < 3.10, you would want to do - from metaflow._vendor.importlib_metadata ..
There was a problem hiding this comment.
using the vendored importlib won't actually solve the issue for older python versions, as editable installs with python <=3.8 do not install dist files, instead producing an .egg-link which is not supported by the importlib metadata.
the difference in install files is most likely due to the setuptools versions that the different python version picks. I can either introduce separate logic for <=3.8 that reads the editable path from the egg-link file, or we leave older versions affected. 3.9 onward works as expected with the default importlib so no need to use vendored one for these
There was a problem hiding this comment.
at least a check that fails nicely would be great
There was a problem hiding this comment.
added a detailed error for the editable lookup failure.
while testing this, some of my virtualenvs were installing metaflow with an egg-link, while others were generating the correct dist metadata for lookup. Tried versions 3.8->3.12
The culprit here seems to be the version of pip. 23.0.1 which is the default for many venvs, installs via egg-link, while pip install pip --upgrade bumps this to 25.0.1 which installs dist metadata on all of the tested python versions.
metaflow-dev upto find the relevant Makefile when metaflow installed with--editableAWS_PROFILEenv var that interferes with the local minio profiles set up bymetaflow-dev shell