As documented in an open pull request for pip-tools:
The dependencies of a package can change depending on the Python environment in which it is installed. Here, we define a Python environment as the combination of Operating System, Python version (2.7, 3.6, etc.), and Python implementation (CPython, PyPy, etc.).
As the resulting
requirements.txt
can differ for each environment, users must executepip-compile
on each Python environment separately to generate arequirements.txt
valid for each said environment. The samerequirements.in
can be used as the source file for all environments, using PEP 508 environment markers as needed, the same way it would be done for regularpip
cross-environment usage.
Based on this, my understanding is, given an application that:
-
Is in the process of migrating from Python 2 to Python 3
-
Will simultaneously support for both environments
-
Has been using layered requirements (i.e., development built on base/production) in Python 2:
├── requirements │ ├── base.in │ ├── base.txt │ ├── dev.in <-- Includes base.txt │ ├── dev.txt
Then the resulting "matrix" of frozen requirements files should be something like:
├── requirements
│ ├── base.in
│ ├── base.txt
│ ├── base-py37.txt
│ ├── dev.in
│ ├── dev.txt
│ ├── dev-py37.txt
I compared Tox and Nox, using a subset of requirements from an existing project. Tox is what "everyone" uses, but since it doesn't yet support parametrization of arbitrary environments, I ended up with multiple configuration files and a hack to determine the output filename. Nox makes it easier to maintain one configuration file, written in Python, but it's not as "battle-tested".
Install Tox and Nox for the current user using pipx:
$ pipx install tox
$ pipx install nox
Recompile requirements in all environments:
$ tox -c requirements
$ nox -s requirements
Upgrade a primary dependency:
- Add a minimal version specifier to a
requirements/*.in
- e.g.
flake8>=3.7
inrequirements/dev.in
- e.g.
- Recompile requirements
Transitive dependencies should be upgraded as needed when a primary dependency is upgraded, but if one needs to be upgraded independently:
- Add it to a
requirements/*.in
- OR remove it from all
requirements/*.txt
- Recompile requirements
Run the test suite in all environments:
$ tox
$ nox
Run the test suite in the Python 2.7 environment:
$ tox -e py27
$ nox -s test-2.7
List outdated dependencies in all environments:
$ tox -- pip list -o
$ nox -- pip list -o
From diff -u dev.txt dev-py37.txt
:
--- dev.txt 2019-05-29 11:42:42.000000000 -0400
+++ dev-py37.txt 2019-05-29 11:42:42.000000000 -0400
@@ -2,27 +2,25 @@
# This file is autogenerated by pip-compile
# To update, run:
#
-# pip-compile --output-file=dev.txt base.txt dev.in
+# pip-compile --output-file=dev-py37.txt base-py37.txt dev.in
#
attrs==19.1.0
certifi==2019.3.9
chardet==3.0.4
-configparser==3.7.4 # via flake8
-enum34==1.1.6 # via flake8
+defusedxml==0.6.0
flake8==3.5.0
-functools32==3.2.3.post2
idna==2.8
jsonschema==3.0.1
mccabe==0.6.1 # via flake8
numpy==1.16.3
oauthlib==3.0.1
-pandas==0.19.2 ; python_version == "2.7"
+pandas==0.24.2 ; python_version > "3"
pycodestyle==2.3.1 # via flake8
pyflakes==1.6.0 # via flake8
pyjwt==1.7.1
pyrsistent==0.15.2
python-dateutil==2.8.0
-python-openid==2.2.5
+python3-openid==3.1.0
pytz==2019.1
requests-oauthlib==1.2.0
requests==2.22.0