Skip to content

Core CLI argument parsing is overly fragile and breaks under common usage scenarios #10748

@pietakio

Description

@pietakio

Current Behavior

For inexplicable reasons, conda only conditionally exposes the activate subcommand when the active Python process satisfies certain abstruse, non-intuitive, and non-trivially debuggable conditions. Specifically, the conda.cli.main.main() functions appears to be conditionally enabling the activate subcommand only when the following ad-hoc argument parsing detection succeeds:

    args = tuple(ensure_text_type(s) for s in args)

    if len(args) > 1:
        try:
            argv1 = args[1].strip()
            if argv1.startswith('shell.'):
                from ..activate import main as activator_main
                return activator_main()
            elif argv1.startswith('..'):
                import conda.cli.activate as activate
                activate.main()
                return

Critically, that detection fails under common usage scenarios. To compound matters, the error message reported when this occurs is unhelpfully ambiguous and fails to actually describe the underlying issue. Specifically, conda activate meaninglessly fails...

  • When preceded by the command shell builtin (which excludes shell aliases from consideration, so I almost always precede everything in all of my shell scripts with that builtin):
$ command conda activate
    CommandNotFoundError: Your shell has not been properly configured to use 'conda %(command)s'.
    To initialize your shell, run
    
        $ conda init <SHELL_NAME>
    
    Currently supported shells are:
      - bash
      - fish
      - tcsh
      - xonsh
      - zsh
      - powershell
    
    See 'conda init --help' for more information and options.
    
    IMPORTANT: You may need to close and restart your shell after running 'conda init'.
  • When run as a Python module rather shell command:
$ python3 -m conda activate
    CommandNotFoundError: Your shell has not been properly configured to use 'conda %(command)s'.
    To initialize your shell, run
    
        $ conda init <SHELL_NAME>
    
    Currently supported shells are:
      - bash
      - fish
      - tcsh
      - xonsh
      - zsh
      - powershell
    
    See 'conda init --help' for more information and options.
    
    IMPORTANT: You may need to close and restart your shell after running 'conda init'.

Of course, everything behaves as expected when running $ conda activate. That's bad, though. Why? Because both the $ command conda activate and $ python3 -m conda activate commands are less ambiguous than the $ conda activate command (which is ambiguous with respect to both shell aliases and the Python interpreter). I expect disambiguous commands to succeed – not unconditionally fail.

Ultimately, all of this is happening because conda is subverting sane Python and POSIX standards. Specifically:

Ad-hoc Argument Parsing

conda is attempting to (...but failing, of course) perform its own ad-hoc argument parsing.

That's a fundamentally bad idea. If you're going to use the stdlib argparse module, just do that. If you're going to perform your own unique hand-rolled brand of argument parsing that violates POSIX expectations, just do that.

But whatever you do, don't first preparse the argument list with non-standard and non-portable argument detection routines that violate POSIX expectations and then try to use the stdlib argparse module. Generally speaking, that's not going to work. And, indeed, that doesn't work here.

No Exception Tracebacks

conda is failing to emit tracebacks. That only compounds matters, because it prevents users from reporting or self-diagnosing the exact context that produced an ambiguous error. You need to let us help you.

To identify the underlying issue here, I actually had to explicitly inject a raise ValueError(message) at the tail end of the CommandNotFoundError.__init__() exception class constructor, which then yielded a usable traceback directly implicating the actual argument handling issue:

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/pietakio/py/conda/lib/python3.9/argparse.py", line 1851, in parse_known_args
        namespace, args = self._parse_known_args(args, namespace)
      File "/home/pietakio/py/conda/lib/python3.9/argparse.py", line 2063, in _parse_known_args
        stop_index = consume_positionals(start_index)
      File "/home/pietakio/py/conda/lib/python3.9/argparse.py", line 2019, in consume_positionals
        take_action(action, args)
      File "/home/pietakio/py/conda/lib/python3.9/argparse.py", line 1912, in take_action
        argument_values = self._get_values(action, argument_strings)
      File "/home/pietakio/py/conda/lib/python3.9/argparse.py", line 2453, in _get_values
        self._check_value(action, value[0])
      File "/home/pietakio/py/conda/lib/python3.9/argparse.py", line 2500, in _check_value
        raise ArgumentError(action, msg % args)
    argparse.ArgumentError: argument command: invalid choice: 'activate' (choose from 'clean', 'compare', 'config', 'create', 'help', 'info', 'init', 'install', 'list', 'package', 'remove', 'uninstall', 'run', 'search', 'update', 'upgrade')
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/pietakio/py/conda/lib/python3.9/site-packages/conda/exceptions.py", line 1080, in __call__
        return func(*args, **kwargs)
      File "/home/pietakio/py/conda/lib/python3.9/site-packages/conda/cli/main.py", line 72, in _main
        args = p.parse_args(args[1:])
      File "/home/pietakio/py/conda/lib/python3.9/argparse.py", line 1818, in parse_args
        args, argv = self.parse_known_args(args, namespace)
      File "/home/pietakio/py/conda/lib/python3.9/argparse.py", line 1854, in parse_known_args
        self.error(str(err))
      File "/home/pietakio/py/conda/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 147, in error
        raise CommandNotFoundError(cmd)
      File "/home/pietakio/py/conda/lib/python3.9/site-packages/conda/exceptions.py", line 304, in __init__
        raise ValueError(message)
    ValueError: Your shell has not been properly configured to use 'conda %(command)s'.
    To initialize your shell, run
    
        $ conda init <SHELL_NAME>
    
    Currently supported shells are:
      - bash
      - fish
      - tcsh
      - xonsh
      - zsh
      - powershell
    
    See 'conda init --help' for more information and options.
    
    IMPORTANT: You may need to close and restart your shell after running 'conda init'.

That's significantly more useful than the current conda approach of silently squelching tracebacks. The argparse.ArgumentError: argument command: invalid choice: 'activate' exception message currently hidden by conda demonstrates that conda isn't actually adding the activate subcommand to its argparse-based argument parsers under the common usage scenarios listed above, which then allowed me to diagnose the actual issue.

...but I shouldn't have had to do that. conda should already be printing, logging, or otherwise emitting tracebacks out-of-the-box – because there's no disadvantage and considerable advantage to doing so.

We're all Pythonistas here. No one's afraid of tracebacks, guys.

Steps to Reproduce

See above, please.

Expected Behavior

The conda activate subcommand should just work regardless of how conda is invoked. Ideally, that subcommand should always be available like every other subcommand.

Pragmatically, attempting to decide whether or not that subcommand should be made available appears to be an undecidable problem. For every additional edge case you add to the ad-hoc argument parsing routine above, there will (probably) exist yet another edge case you haven't considered that will eventually break that routine for someone somewhere under a context you failed to consider.

So stop considering the context. Just always add conda activate.

Environment Information

conda info
     active environment : base
    active env location : /home/pietakio/py/conda
            shell level : 1
       user config file : /home/pietakio/.condarc
 populated config files : /home/pietakio/.condarc
          conda version : 4.10.1
    conda-build version : not installed
         python version : 3.9.1.final.0
       virtual packages : __cuda=11.2=0
                          __linux=5.4.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/pietakio/py/conda  (writable)
      conda av data dir : /home/pietakio/py/conda/etc/conda
  conda av metadata url : https://repo.anaconda.com/pkgs/main
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/pietakio/py/conda/pkgs
                          /home/pietakio/.conda/pkgs
       envs directories : /home/pietakio/py/conda/envs
                          /home/pietakio/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.1 requests/2.25.1 CPython/3.9.1 Linux/5.4.0-77-generic ubuntu/20.04.2 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
conda config --show-sources
==> /home/pietakio/.condarc <==
channels:
  - conda-forge
  - defaults
report_errors: True
conda list --show-channel-urls
# packages in environment at /home/pietakio/py/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             4.5                       1_gnu    defaults
brotlipy                  0.7.0           py39h27cfd23_1003    defaults
ca-certificates           2021.5.25            h06a4308_1    defaults
certifi                   2021.5.30        py39h06a4308_0    defaults
cffi                      1.14.5           py39h261ae71_0    defaults
chardet                   4.0.0           py39h06a4308_1003    defaults
conda                     4.10.1           py39h06a4308_1    defaults
conda-package-handling    1.7.3            py39h27cfd23_1    defaults
cryptography              3.4.7            py39hd23ed53_0    defaults
idna                      2.10               pyhd3eb1b0_0    defaults
ld_impl_linux-64          2.35.1               h7274673_9    defaults
libffi                    3.3                  he6710b0_2    defaults
libgcc-ng                 9.3.0               h5101ec6_17    defaults
libgomp                   9.3.0               h5101ec6_17    defaults
libstdcxx-ng              9.3.0               hd4cf53a_17    defaults
ncurses                   6.2                  he6710b0_1    defaults
openssl                   1.1.1k               h27cfd23_0    defaults
pycosat                   0.6.3            py39h27cfd23_0    defaults
pycparser                 2.20                       py_2    defaults
pyopenssl                 20.0.1             pyhd3eb1b0_1    defaults
pysocks                   1.7.1            py39h06a4308_0    defaults
python                    3.9.1                hdb3f193_2    defaults
readline                  8.1                  h27cfd23_0    defaults
requests                  2.25.1             pyhd3eb1b0_0    defaults
ruamel_yaml               0.15.100         py39h27cfd23_0    defaults
setuptools                52.0.0           py39h06a4308_0    defaults
six                       1.16.0             pyhd3eb1b0_0    defaults
sqlite                    3.35.4               hdfb4753_0    defaults
tk                        8.6.10               hbc83047_0    defaults
tqdm                      4.61.1             pyhd3eb1b0_1    defaults
tzdata                    2021a                h52ac0ba_0    defaults
urllib3                   1.26.4             pyhd3eb1b0_0    defaults
xz                        5.2.5                h7b6447c_0    defaults
yaml                      0.2.5                h7b6447c_0    defaults
zlib                      1.2.11               h7b6447c_3    defaults

Sorry for the Grumps

This issue pains me, because our team otherwise loves and adores conda. The conda ecosystem is the future of Python packaging, because Python packaging is otherwise hell. For the collective sanity of the burgeoning (Anaconda|Miniconda|Bomba) community, it'd be great if we could get this right once and for all.

Thanks for all the continued hard work, everyone! 😍

Tasks

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogissue has been triaged but has not been earmarked for any upcoming releaseclipertains to the CLI interfaceduplicateindicate issues/PRs that are duplicates of anotherlocked[bot] locked due to inactivityplugins::activatepertains to conda-activate or conda-deactivatesource::communitycatch-all for issues filed by community memberstype::tech-debtidentifies or resolves some technical debt

    Type

    No type

    Projects

    Status

    🏁 Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions