Skip to content

Improve layered dependency workflow #1092

@MartinAltmayer

Description

@MartinAltmayer

What's the problem this feature will solve?

I know this has been discussed a few times (e.g. #398), but the current solution for layered dependencies still doesn't work in all cases. For example the following simple files cannot be compiled:

# base.in
requests

# dev.in
-c base.txt
moto

Running pip-compile base.in results in

certifi==2019.11.28       # via requests
chardet==3.0.4            # via requests
idna==2.9                 # via requests
requests==2.23.0          # via -r base.in
urllib3==1.25.8           # via requests

Now pip-compile dev.in aborts with an error:

Could not find a version that matches idna<2.9,==2.9,>=2.5 (from -c base.txt (line 9))
Tried: 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 2.0, 2.0, 2.1, 2.1, 2.2, 2.2, 2.3, 2.3, 2.4, 2.4, 2.5, 2.5, 2.6, 2.6, 2.7, 2.7, 2.8, 2.8, 2.9, 2.9
There are incompatible versions in the resolved dependencies:
  idna==2.9 (from -c base.txt (line 9))
  idna<2.9,>=2.5 (from moto==1.3.14->-r dev.in (line 2))

A solution exists, though: Just use idna==2.8.

Describe the solution you'd like

In my opinion, a solution to the layered dependencies problem requires to compile the outermost layer first: Then the resolver has a chance to find a solution for all dependencies. A working solution is the following:

# base.in
requests

# dev.in
-r base.in
moto

# constraint.in
-c dev.txt

Now I can generate correct dependency sets with

pip-compile dev.in
pip-compile base.in constraint.in --output-file base.txt

The constraint.in is necessary because otherwise the second pip-compile will generate versions that differ from those in dev.txt.

What do you think of this solution? How about adding a -c/--constrain option to pip-compile, that acts like I added a constraint.in file above. Then I could run

pip-compile dev.in
pip-compile -c dev.txt base.in

with the same effect as above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    cliRelated to command line interface thingsfeatureRequest for a new featureresolverRelated to dependency resolver

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions