The only potential gotcha here for users that I think is good to call out here is compatibility with other dependencies that depend on qiskit-terra and/or qiskit<1. I think this will trip up users the most more than not being able to pip install -U qiskit because it's the piece that requires understanding how pip does dep solving or lack thereof and people doing things like python -m venv test && test/bin/pip install qiskit>=1.0 && test/bin/pip install qiskit-qubit-reuse (this is a good reminder I should update the plugin package to not depend on terra before 1.0). This will end up in a mixed installation state between qiskit 1.0.0 and qiskit-terra 0.46.0. I think this really should be called out in the top and not just in the troubleshooting guide below.

I've thrown in an admonition warning about this, including a link to the troubleshooting section if people do get into problems in 75f6e3a. Do you think it's enough?

I don't necessarily want to get too pessimistic up at the top of the guide - if they're just using IBM tooling (which I imagine most users are), they're hopefully not going to have any issues by the time 1.0 is actually released.

I tried to make this a proper HTML <dl> definition list, but we don't seem to have styling for that, so it ends up very sad:

The bullets here aren't ideal, but better than whatever ^ that is.

I can reformat this into a table if that would seem more appropriate? terms in the left column, defs in the right?

I'm happy with whatever styling you think would be best. My guess is that a table might look a bit unbalanced and hard to read by squeezing the "definition" paragraphs a bit much, but I don't know and I'm fine deferring to you guys in design anyway.

Perhaps here you can link to the section on how to use pipdeptree to find dependency versions.

I put in an extra comment and link in f4b16f0, though I avoided dropping an actual link to the specific troubleshooting section below, because that's all about recovering from a broken environment. I don't actually know a simple way to get pip (or pipdeptree) to tell you the requirements of a particular prospective package without attempting to install it (though maybe the experimental pip index will eventually do that).

I don't know if anyone is reading this far down in the guide, but I think the problem could also be __pycache__ directories if you use pip install -e. You could suggest git clean -fxd after the user checks that there are no important uncommitted files.

__pycache__ shouldn't cause problems here, because the first importlib search is for distributions not packages, and then the second is a search for "is qiskit.tools a loadable module with a backing source file?". It's not enough for the qiskit/tools directory to be on the path, even if qiskit/tools/__pycache__/__init__.*.pyc exists, because we test that the ModuleSpec has a proper location for the file (qiskit/tools.py or qiskit/tools/__init__.py) to verify it's not an empty namespace package left over from a dirty source tree.

I'm hesitant to suggest git clean for general use because it's too easy to accidentally take out local changes, or clean out all your cached tox environments, etc.

My comment was misleading. When I said "this problem," I was thinking of the problem of left over files in a repo that had been installed in editable mode rather than the specific import qiskit problem with left over namespace package metadata.

I didn't fully test it but somehow there were some code removals that didn't trigger import errors for me with my editable install, and I had assumed the imports were still working because that is how things worked with Python 2 and the .pyc files left over in directories that had had the .py files removed. Maybe it was actually the info directories or something else. I didn't try to recreate the problem after doing the clean.

That particular case was coming up for me with other packages that set up Qiskit plugins in the package metadata (specifically qiskit-ibm-provider's transpiler pass plugin that imported from qiskit.extensions). It kept working in my editable install but then failed in the clean CI environment. It is a surprising case because I was not using the plugin or anything else from qiskit-ibm-provider but just having the distribution with the plugin installed is enough to trigger an error when trying to do a transpile. I was wondering if there was anything to mention related to that in the troubleshooting section, but I think ultimately it is a consequence of having a package that is not compatible with Qiskit 1.0 installed, so that is already covered.

Could you send me a set of repro steps, and I'll look into either making the import check smarter, or adding an extra bit to the troubleshooting section in this guide in a day or so, after an initial version is public?

I like all the detail here. I wonder if there should be a section (maybe linkable) that just says that the files that were in qiskit-terra are now in qiskit and that pip does not know how to handle overlapping files and will overwrite and erase files from qiskit when operating on qiskit-terra and vice versa, as the quick version of why the careful steps described in the user section are necessary and then those interested can expand out from there.

I think that's an explanation of what about the environment is actually broken, more than "why we did this", maybe? Do you think we could leave that out, or at least add it a bit later? (I'm keen to get this merged and published.)

Right, I just suggested it here because I had originally suggested it up above before getting to this section and seeing that you had largely covered all the details of what was going on with packaging here. I'll leave it up to you. I don't know what is best to tell users. Mainly what I was thinking is that if a user understood about the same file paths being in both packages and pip not trying to avoid conflicts it might give intuition for understanding problems that arise.