pip-sync yields different results depending on pre-installed package set #896
Description
Update: The shape of this issue has changed a bit since creation, due to discoveries in this discussion, and the addition of the --pip-args option. I'm renaming it, and adding a new summary:
If pip-sync has to actively install a requirement, its dependencies will end up in the environment. But if the requirement is already installed when pip-sync runs, its dependencies not present in the txt will be removed, "breaking" the environment.
$ echo "pytest==5.1.2" > requirements.txt
$ pip-sync /dev/null
$ pip-sync requirements.txt
$ pip freeze | wc -l
12
$ pip-sync requirements.txt
$ pip freeze | wc -l
4A txt which omits dependencies is, by pip-tools standards, malformed, and therefore pip-sync's behavior is not currently guaranteed or defined. But:
IMO "running pip-sync a second time makes no further changes" is a very important property to maintain
-
IMO a package is not "properly" installed if its dependencies aren't present, and
pip-syncshould achieve "proper" installs. -
If a user wants to "break" dependencies, they can explicitly do so with
--pip-args --no-deps -
While I don't see an advantage to the current (inconsistent) behavior, having the deps always installed (unless
--no-depsis specified) would enable users to usepip-syncwithout issue ontxts that don't come frompip-compile. Though not officially supported, it's a nice bonus to be able to usepip-syncuniformly as a complete replacement/wrapper forpip install -r. Now that Exclude requirements with non-matching markers from pip-sync's merge stage #927 is merged, I don't know of any other obstacle to supporting "foreign"txts (in practice, if not policy).
My idea of a solution is #907, which keeps already-installed packages in the to-install set, as pip itself will refrain from reinstalling those anyway, and will ensure its dependencies get installed if necessary.
Original report:
I looked but didn't find an existing issue for this.
What's the problem this feature will solve?
I would like to use pip-sync on requirements.txt files that are not generated by pip-compile, for arbitrary (other folks') Python projects. In fact I had been doing so, and I guess I have been getting lucky, as I only recently came to a case where this failed, and was surprised to find when re-reading pip-tools's README:
Be careful: pip-sync is meant to be used only with a requirements.txt generated by pip-compile
The failure I notice is that pip-sync will uninstall hard dependencies of packages listed in the requirements.txt. I don't know if there are further problems.
Describe the solution you'd like
Until/unless there are other specifically identified problems for externally crafted requirements.txt files, the solution would seem to be for pip-sync to refrain from uninstalling dependencies of explicitly listed packages. This might be as simple as doing all the uninstalling before any of the installing.
This would enable pip-tools users to continue to use pip-sync across all Python projects they work on, even if they are not in a position to enforce use of pip-compile to manage those projects' requirements.txt generation.
Alternative Solutions
Alternatively, of course, users can just use pip install -r requirements.txt. That however does not have the advantage of stripping the environment of non-required packages. Although I guess the solution could then be as simple as pip-sync requirements.txt && pip install -r requirements.txt.
Additional context
Simple demonstration of the problem:
echo "pytest==5.1.2" > requirements.txt
pip install -r requirements.txt
pip freezeatomicwrites==1.3.0
attrs==19.1.0
Click==7.0
importlib-metadata==0.22
more-itertools==7.2.0
packaging==19.1
pip-tools==4.1.0
pluggy==0.13.0
py==1.8.0
pyparsing==2.4.2
pytest==5.1.2
six==1.12.0
wcwidth==0.1.7
zipp==0.6.0
pip-sync requirements.txt
pip freezeClick==7.0
pip-tools==4.1.0
pytest==5.1.2
six==1.12.0
The result is a broken installation of pytest.