Skip to content

Conversation

itsayellow
Copy link
Contributor

  • I have added an entry to docs/changelog.md

Summary of changes

The main purpose of this is to address the following issues: #627 , #390 , #109 . It adds a --json switch to pipx list which directs pipx list to output rich information about all the venvs in json format. The output is suitable to be directed to a file, e.g. pipx list --json > pipx.json.

The implementation basically dumps the contents of every venv's pipx_metadata.json file into an object that is then dumped into the output. As such, there is probably more information than is strictly necessary to reinstall these packages.

The json produced follows the following structure:

{
    "pipx_spec_version": "0.1",
    "venvs": {
        "<venv_name1>": {
            "metadata": <contents of pipx_metadata.json for venv_name1>
        },
        "<venv_name2>": {
            "metadata": <contents of pipx_metadata.json for venv_name2>
        },
        [...]
    }
}

Here is an example of a sample pipx listing, both with --json and without, for pipx-installed pycowsay, and black with injected pylint:

>  pipx list --json
{
    "pipx_spec_version": "0.1",
    "venvs": {
        "black": {
            "metadata": {
                "injected_packages": {
                    "pylint": {
                        "app_paths": [
                            {
                                "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/epylint",
                                "__type__": "Path"
                            },
                            {
                                "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/pylint",
                                "__type__": "Path"
                            },
                            {
                                "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/pyreverse",
                                "__type__": "Path"
                            },
                            {
                                "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/symilar",
                                "__type__": "Path"
                            }
                        ],
                        "app_paths_of_dependencies": {
                            "isort": [
                                {
                                    "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/isort",
                                    "__type__": "Path"
                                },
                                {
                                    "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/isort-identify-imports",
                                    "__type__": "Path"
                                }
                            ]
                        },
                        "apps": [
                            "epylint",
                            "pylint",
                            "pyreverse",
                            "symilar"
                        ],
                        "apps_of_dependencies": [
                            "isort",
                            "isort-identify-imports"
                        ],
                        "include_apps": false,
                        "include_dependencies": false,
                        "package": "pylint",
                        "package_or_url": "pylint",
                        "package_version": "2.7.2",
                        "pip_args": [],
                        "suffix": ""
                    }
                },
                "main_package": {
                    "app_paths": [
                        {
                            "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/black",
                            "__type__": "Path"
                        },
                        {
                            "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/black-primer",
                            "__type__": "Path"
                        },
                        {
                            "__Path__": "/Users/mclapp/.local/pipx/venvs/black/bin/blackd",
                            "__type__": "Path"
                        }
                    ],
                    "app_paths_of_dependencies": {},
                    "apps": [
                        "black",
                        "black-primer",
                        "blackd"
                    ],
                    "apps_of_dependencies": [],
                    "include_apps": true,
                    "include_dependencies": false,
                    "package": "black",
                    "package_or_url": "black",
                    "package_version": "20.8b1",
                    "pip_args": [],
                    "suffix": ""
                },
                "pipx_metadata_version": "0.2",
                "python_version": "Python 3.9.2",
                "venv_args": []
            }
        },
        "pycowsay": {
            "metadata": {
                "injected_packages": {},
                "main_package": {
                    "app_paths": [
                        {
                            "__Path__": "/Users/mclapp/.local/pipx/venvs/pycowsay/bin/pycowsay",
                            "__type__": "Path"
                        }
                    ],
                    "app_paths_of_dependencies": {},
                    "apps": [
                        "pycowsay"
                    ],
                    "apps_of_dependencies": [],
                    "include_apps": true,
                    "include_dependencies": false,
                    "package": "pycowsay",
                    "package_or_url": "pycowsay",
                    "package_version": "0.0.0.1",
                    "pip_args": [],
                    "suffix": ""
                },
                "pipx_metadata_version": "0.2",
                "python_version": "Python 3.9.2",
                "venv_args": []
            }
        }
    }
}
> pipx list --include-injected
venvs are in /Users/mclapp/.local/pipx/venvs
apps are exposed on your $PATH at /Users/mclapp/.local/bin
   package black 20.8b1, Python 3.9.2
    - black
    - black-primer
    - blackd
    Injected Packages:
      - pylint 2.7.2
   package pycowsay 0.0.0.1, Python 3.9.2
    - pycowsay

Additionally I factored out the code that checks for venv problems into venv_health_check().

Test plan

Tested by running

pipx list --json

@itsayellow
Copy link
Contributor Author

I'll add tests for this if everybody think the approach is good.

@itsayellow itsayellow marked this pull request as ready for review April 2, 2021 02:29
@itsayellow itsayellow requested a review from a team April 3, 2021 17:51
@itsayellow
Copy link
Contributor Author

Looks like everybody is busy 😉 @cs01, does this seem like a basically good idea? Any thoughts?

@cs01
Copy link
Member

cs01 commented Apr 13, 2021

Sorry. Yes this looks good. Thanks for adding this. I have been thinking about it for a long time but never did it. You made it happen though 👍

@itsayellow itsayellow merged commit 8a7c1da into pypa:master Apr 13, 2021
@itsayellow itsayellow deleted the list-json branch April 13, 2021 21:02
@liskin
Copy link

liskin commented Apr 28, 2021

This is awesome! ♥

Unfortunately there's a little issue still: initially, when nothing is installed, pipx list --json prints nothing has been installed with pipx 😴 to stdout, and that's not valid json.

@itsayellow
Copy link
Contributor Author

This is awesome! ♥

Unfortunately there's a little issue still: initially, when nothing is installed, pipx list --json prints nothing has been installed with pipx 😴 to stdout, and that's not valid json.

Dang it :( . OK, need to fix that...thanks for reporting!

@gwerbin
Copy link

gwerbin commented May 16, 2021

This is great!

I took a rough pass at defining a JSON Schema for this metadata (note the hypothetical URL for the $id field): https://gist.github.com/gwerbin/6f6d6085e63bc4e4168b82c23d60439e

Such a schema could be hosted on the Pipx website, and/or provided with the Pipx application itself. You could use it internally for your test suite, as well as to allow other tools to develop against or reuse the Pipx JSON format with confidence.


Also for the TOML and YAML lovers out there, you can convert between formats with:

pipx list --json | python -c 'import sys,json,yaml; yaml.dump(json.load(sys.stdin), sys.stdout)' > pipx.toml
python -c 'import sys,json,yaml; print(json.dumps(yaml.safe_load(sys.stdin), indent=2))' < pipx.toml

pipx list --json | python -c 'import sys,json,toml; toml.dump(json.load(sys.stdin), sys.stdout)' > pipx.toml
python -c 'import sys,json,toml; print(json.dumps(toml.load(sys.stdin), indent=2))' < pipx.toml

You can even add it to a Makefile if you wanted:

pipx.json:
	pipx list --json > $@
pipx.toml: pipx.json
	python -c 'import sys,json,toml; toml.dump(json.load(sys.stdin), sys.stdout)' < $^ > $@
pipx.yaml: pipx.json
	python -c 'import sys,json,yaml; yaml.dump(json.load(sys.stdin), sys.stdout)' < $^ > $@

# Other targets can now depend on pipx.json, pipx.toml, and/or pipx.yaml

Note that there could be problems round-tripping with TOML, since the JSON format is a bit more complicated than what you'd usually write in a TOML file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants