What's the problem this feature will solve?

pip-tools currently handles paths in general a bit weakly:

• using string parsing and manipulation, achieving incorrect results depending on the path names, potentially overwriting unrelated user files
• producing different compilation output depending on the form given for equivalent relative paths (./requirements.in vs requirements.in)
• naively forwarding relative paths (relative to a reqsin file) into the output file annotations, which may be in a different folder
• not yet annotating setup.py sources with relative paths if appropriate
• occasionally issues are raised about transformation between absolute and relative paths for various purposes
• sometimes flops when trying to write to un-writable cache paths

We need to reliably inspect and transform paths, absolute and relative, for issues present and future.

Describe the solution you'd like

I'd like to build a consensus on a high level approach to more robust and consistent path handling (which modules to use and how it will affect the packaging of pip-tools itself), and then use that approach to fix up and improve path operations.

Here's an example problem in the current code:

$ echo '# real important data' > some.txt
$ mkdir some.folder
$ echo 'requests' > some.folder/reqsin
$ pip-compile --no-header some.folder/reqsin
$ ls some.folder
reqsin
$ cat some.txt
certifi==2019.11.28       # via requests
chardet==3.0.4            # via requests
idna==2.8                 # via requests
requests==2.22.0          # via -r some.folder/reqsin (line 1)
urllib3==1.25.8           # via requests

Similarly, if some.txt content is uncommented, pip-compile fails because it tries to parse that for existing requirements.

See also: #1061 #204 #966 #616 #395 #1084 and discussion at #1058

Alternatives and Considerations

• We're still supporting python2
• pathlib (stdlib) + pathlib2 (pypi or vendored)
• better use of os.path
• plumbum (which could also supplant click)