Skip to content

Conversation

jezdez
Copy link
Member

@jezdez jezdez commented Mar 7, 2023

Description

Resolves #12455.

Checklist - did you ...

  • Add a file to the news directory (using the template) for the next release's release notes?
  • Add / update necessary tests?
  • Add / update outdated documentation?

@conda-bot conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Mar 7, 2023
caplog.set_level(logging.INFO, logger="conda.plugins.manager")
plugin_manager.load_setuptools_entrypoints("conda")
assert plugin_manager.get_plugins() == set()
assert "load_setuptools_entrypoints ImportError" in caplog.text
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I bet you could add an ImportError in a different way. See https://github.com/conda/conda/blob/main/tests/gateways/test_repodata_gateway.py#L113 or

mocker.patch.dict("sys.modules", values=modules, clear=True)

@dholth
Copy link
Contributor

dholth commented Mar 7, 2023

The other thing we might want to do is audit our plugins for too-eager imports. Ideally scanning the entry point would avoid importing most of the plugin implementation.

@jezdez
Copy link
Member Author

jezdez commented Mar 7, 2023

The other thing we might want to do is audit our plugins for too-eager imports. Ideally scanning the entry point would avoid importing most of the plugin implementation.

I'm not following, scanning the entry point?

@dholth
Copy link
Contributor

dholth commented Mar 7, 2023

When a plugin is imported, just to be added to the list of conda x, it should not do very much work. Including importing things.

A plugin author might want to arrange a plugin with separate folders myplugin/cli and myplugin/implementation so that "import myplugin.cli" doesn't import whatever dependencies "myplugin.implementation" uses to do its thing. Inside "myplugin.cli:do_the_plugin()", import "myplugin.implementation".

Then unrelated commands remain fast.

Firefox used to say "these extensions are slowing down your browser" and tell you how long each one took to load. Then you could decide whether to continue to use those extensions. It also has a safe mode to avoid loading all extensions.

@jezdez
Copy link
Member Author

jezdez commented Mar 7, 2023

I see, lazy loading plugins, but I don't see how this works with pluggy as it uses importlib as the central system to load entrypoints based plugins which loads plugins with the import system. I'm not sure how to achieve this given these constraints. I'll take a look at it tomorrow.

@dholth
Copy link
Contributor

dholth commented Mar 7, 2023

It depends on the type of the plugin.

For conda_subcommands it is simple. Image some subcommand uses sqlalchemy.

A. someplugin.py

import sqlalchemy
def custom_command():
    sqlalchemy.do_a_thing()

@plugins.hookimpl
def conda_subcommands(self):
    yield CondaSubcommand(
        name="custom",
        summary="test custom command",
        action=self.custom_command,
    )

B. anotherplugin.py

def custom_command():
    import sqlalchemy
    sqlalchemy.do_a_thing()

@plugins.hookimpl
def conda_subcommands(self):
    yield CondaSubcommand(
        name="custom",
        summary="test custom command",
        action=self.custom_command,
    )

The second plugin is faster.

Authors would be more likely to do this accidentally, by having an __init__.py that imports a bunch of things sitting between your entry points hook and the @plugins.hookimpl call.

@jezdez
Copy link
Member Author

jezdez commented Mar 8, 2023

It depends on the type of the plugin.

For conda_subcommands it is simple. Image some subcommand uses sqlalchemy.

A. someplugin.py

import sqlalchemy
def custom_command():
    sqlalchemy.do_a_thing()

@plugins.hookimpl
def conda_subcommands(self):
    yield CondaSubcommand(
        name="custom",
        summary="test custom command",
        action=self.custom_command,
    )

B. anotherplugin.py

def custom_command():
    import sqlalchemy
    sqlalchemy.do_a_thing()

@plugins.hookimpl
def conda_subcommands(self):
    yield CondaSubcommand(
        name="custom",
        summary="test custom command",
        action=self.custom_command,
    )

The second plugin is faster.

Authors would be more likely to do this accidentally, by having an __init__.py that imports a bunch of things sitting between your entry points hook and the @plugins.hookimpl call.

Yeah, I get that, import-time side effects have a downside. For subcommands we could allow to optionally pass a dotted import path as the action parameter and lazy-import it when we need it, to move the moment of loading to later in the stack.

@jezdez jezdez marked this pull request as ready for review March 9, 2023 12:34
@jezdez jezdez requested a review from a team as a code owner March 9, 2023 12:34
@jezdez jezdez requested a review from dholth March 9, 2023 12:37
@dholth
Copy link
Contributor

dholth commented Mar 9, 2023

Thanks @jezdez, this is a much better experience than the older behavior.

dholth
dholth previously approved these changes Mar 9, 2023
@jezdez jezdez requested a review from dholth March 10, 2023 18:54
f"Error while loading conda plugins from entrypoints: {err}"
)
count = 0
for dist in list(distributions()):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the retconned "same thing setuptools did" equivalent function?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, importlib.metadata is the pkg_resources equivalent

continue
self.register(plugin, name=entry_point.name)
count += 1
return count
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we get this line to run

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not following, this just returns the count, there is no return statement in the previous code. This contains code from https://github.com/pytest-dev/pluggy/blob/c18d5da87f33bc5359c6eeccf4e85a194842d907/src/pluggy/_manager.py#L352-L377 BTW

### Enhancements

* Handle Python import errors gracefully when loading
conda plugins from entrypoints. (#12460)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solid description

@jezdez jezdez requested a review from dholth March 10, 2023 21:58
@dholth
Copy link
Contributor

dholth commented Mar 10, 2023

Did we fake a working plugin too as long as we're in here?

Copy link
Contributor

@dholth dholth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should help.

@jezdez
Copy link
Member Author

jezdez commented Mar 13, 2023

Did we fake a working plugin too as long as we're in here?

Nah, but it's a good idea to do that in a separate PR.

@jezdez jezdez merged commit 0cac7c4 into main Mar 13, 2023
@jezdez jezdez deleted the plugins-errors branch March 13, 2023 11:28
@jezdez jezdez changed the title Handle ImportError gracefully while loading plugins. Handle exceptions gracefully while loading plugins. May 10, 2023
@github-actions github-actions bot added the locked [bot] locked due to inactivity label May 10, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla-signed [bot] added once the contributor has signed the CLA locked [bot] locked due to inactivity
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Conda should not crash on plugin load "ImportError"
3 participants