Hi Paul, thank you very much for the project.

It was really important for me as a reference when implementing PEP 660 support in setuptools.
I still plan to contribute to editables in terms of namespace handling once the details are flashed out (this may depend on the instructions we get in https://github.com/python/cpython/issues/92054).

I am opening this issue to seek for advice/have a general discussion about using import hooks in editable installs.

Recently in the setuptools repository we got an issue reporting that the editable installation of a regular package (ie. non-namespace) don't behave in the same way as a normal installation.
I understand that editables would also have the same problem. So I wanted to share with you this particular use case and check what are your thoughts.

Problem description

Let's imagine a monorepo that contains multiple directories, each one of them corresponding to a Python project, like the following:

monorepo
├── pkgA
│   ├── pkgA
│   │   └── __init__.py
│   └── pyproject.toml
└── pkgB
    ├── pkgB
    │   └── __init__.py
    └── pyproject.toml

When the users perform a regular installation of one sub-directory, they are able to run scripts importing the installed package; even when the "current working directory (CWD)" is the root of the monorepo.
This is demonstrated in the snippet bellow:

# --- Simulate monorepo project ---
rm -rf /tmp/monorepo
mkdir -p /tmp/monorepo/pkgA/pkgA
mkdir -p /tmp/monorepo/pkgB/pkgB
cd /tmp/monorepo

touch pkgA/pyproject.toml  pkgB/pyproject.toml  
echo "x = 42" > pkgA/pkgA/__init__.py
echo "y = 42" > pkgB/pkgB/__init__.py

# --- Simulate regular installation ---
python3.8 -m venv .venv
cp -r pkgA/pkgA .venv/lib/python3.8/site-packages/
tree .venv/lib/python3.8/site-packages/pkgA
# .venv/lib/python3.8/site-packages/pkgA
# └── __init__.py

# --- Experiment with regular installation ---
.venv/bin/python -c 'from pkgA import x; print(x)'   # <------ this works fine
# 42

.venv/bin/python -c 'import pkgA; print(pkgA)'   # <------ this works fine
# <module 'pkgA' from '/tmp/monorepo/.venv/lib/python3.8/site-packages/pkgA/__init__.py'>

In normal conditions, "stuff" in the CWD should take precedence over "stuff" in site-packages.
However, this rule does not seem to apply for namespace packages in CWD.

If I had to make a wild guess, I would say that the following is happening:

1. the import machinery starts by creating a ModuleSpec for the namespace in CWD
2. half-way through, the import machinery scans site-packages and "changes opinion"
3. the import machinery replaces the initially found namespace ModuleSpec with a new (regular) one pointing to the directory with __init__.py.

When using a import hook/meta path finder, the behaviour differs, and the users are surprised with an error:

# --- Simulate editable installation ---
.venv/bin/python -m pip install -U 'editables==0.3'

rm -rf .venv/lib/python3.8/site-packages/pkgA
echo 'import _editable_impl_pkga' > .venv/lib/python3.8/site-packages/pkga.pth
cat <<EOF > .venv/lib/python3.8/site-packages/_editable_impl_pkga.py
from editables.redirector import RedirectingFinder as F
F.install()
F.map_module('pkgA', '/tmp/monorepo/pkgA/pkgA/__init__.py')
EOF

# --- Experiment with editable installation ---
.venv/bin/python -c 'from pkgA import x; print(x)'   # <------ exception
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
# ImportError: cannot import name 'x' from 'pkgA' (unknown location)

.venv/bin/python -c 'import pkgA; print(pkgA)'   # <------ exception
# <module 'pkgA' (namespace)>

In this scenario the "accidental" namespace package created by having "" on sys.path takes precedence over the import hook.
Since the objective of an editable install is to operate as close as possible to a regular install, this use case could be used as a counter argument for import hooks.