Skip to content

bhrutledge/requirements-py2-py3

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Managing requirements in Python 2 and 3 with pip-tools

As documented in an open pull request for pip-tools:

The dependencies of a package can change depending on the Python environment in which it is installed. Here, we define a Python environment as the combination of Operating System, Python version (2.7, 3.6, etc.), and Python implementation (CPython, PyPy, etc.).

As the resulting requirements.txt can differ for each environment, users must execute pip-compile on each Python environment separately to generate a requirements.txt valid for each said environment. The same requirements.in can be used as the source file for all environments, using PEP 508 environment markers as needed, the same way it would be done for regular pip cross-environment usage.

Based on this, my understanding is, given an application that:

  • Is in the process of migrating from Python 2 to Python 3

  • Will simultaneously support for both environments

  • Has been using layered requirements (i.e., development built on base/production) in Python 2:

    ├── requirements
    │  ├── base.in
    │  ├── base.txt
    │  ├── dev.in    <-- Includes base.txt
    │  ├── dev.txt
    

Then the resulting "matrix" of frozen requirements files should be something like:

├── requirements
│  ├── base.in
│  ├── base.txt
│  ├── base-py37.txt
│  ├── dev.in
│  ├── dev.txt
│  ├── dev-py37.txt

Compiling and using environment-specific requirements

I compared Tox and Nox, using a subset of requirements from an existing project. Tox is what "everyone" uses, but since it doesn't yet support parametrization of arbitrary environments, I ended up with multiple configuration files and a hack to determine the output filename. Nox makes it easier to maintain one configuration file, written in Python, but it's not as "battle-tested".

Install Tox and Nox for the current user using pipx:

$ pipx install tox
$ pipx install nox

Recompile requirements in all environments:

$ tox -c requirements
$ nox -s requirements

Upgrade a primary dependency:

  • Add a minimal version specifier to a requirements/*.in
    • e.g. flake8>=3.7 in requirements/dev.in
  • Recompile requirements

Transitive dependencies should be upgraded as needed when a primary dependency is upgraded, but if one needs to be upgraded independently:

  • Add it to a requirements/*.in
  • OR remove it from all requirements/*.txt
  • Recompile requirements

Run the test suite in all environments:

$ tox
$ nox

Run the test suite in the Python 2.7 environment:

$ tox -e py27
$ nox -s test-2.7

List outdated dependencies in all environments:

$ tox -- pip list -o
$ nox -- pip list -o

Differences between Python 2 and Python 3

From diff -u dev.txt dev-py37.txt:

--- dev.txt     2019-05-29 11:42:42.000000000 -0400
+++ dev-py37.txt        2019-05-29 11:42:42.000000000 -0400
@@ -2,27 +2,25 @@
 # This file is autogenerated by pip-compile
 # To update, run:
 #
-#    pip-compile --output-file=dev.txt base.txt dev.in
+#    pip-compile --output-file=dev-py37.txt base-py37.txt dev.in
 #
 attrs==19.1.0
 certifi==2019.3.9
 chardet==3.0.4
-configparser==3.7.4       # via flake8
-enum34==1.1.6             # via flake8
+defusedxml==0.6.0
 flake8==3.5.0
-functools32==3.2.3.post2
 idna==2.8
 jsonschema==3.0.1
 mccabe==0.6.1             # via flake8
 numpy==1.16.3
 oauthlib==3.0.1
-pandas==0.19.2 ; python_version == "2.7"
+pandas==0.24.2 ; python_version > "3"
 pycodestyle==2.3.1        # via flake8
 pyflakes==1.6.0           # via flake8
 pyjwt==1.7.1
 pyrsistent==0.15.2
 python-dateutil==2.8.0
-python-openid==2.2.5
+python3-openid==3.1.0
 pytz==2019.1
 requests-oauthlib==1.2.0
 requests==2.22.0

About

Managing requirements in Python 2 and 3 with pip-tools

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages