Implement Virtual Packages Plugins #11854

beeankha · 2022-09-16T20:48:29Z

This work will enable virtual packages plugins to be able to hook into conda. This is something with precedented demand:

Plugin mechanism for virtual packages #10131

Fixes #11811

Initial plugin framework implementation. Adds registration decorator `@conda.plugins.register` and hook `conda_subcommands` this allows simple subcommand registration: ``` import conda.plugins @conda.plugins.register def conda_subcommands(): yield conda.plugins.CondaSubcommand( name="my-subcommand", summary="...", action=<function>, ) ``` Co-authored-by: Filipe Lains <lains@riseup.net> Co-authored-by: Katherine Kinnaman <kkinnaman@anaconda.com> Co-authored-by: Jannis Leidel <jannis@leidel.info> Co-authored-by: Travis Hathaway <travis.j.hathaway@gmail.com> Co-authored-by: Ken Odegard <kodegard@anaconda.com>

kenodegard

I'm confused, this implements the hook, but the hook isn't used anywhere? Further, what's the intention of the hook? Will simply installing the plugin set the virtual package as always "on" with a fixed version? E.g., if I'm on macOS I'll always have a corresponding OS package like conda-plugin-osx-10.6 installed?

jezdez · 2022-09-23T11:31:22Z

Yeah, this currently doesn't have the code to make use of the hook with the existing, included virtual packages, that were implemented in e8f9a42. @beeankha Could you look into porting this over to your code please?

* Add `win-arm64` as a recognized platform (conda#11778) * Rename `conda.recipe` to `recipe` (conda#11774) * Rename conda.recipe to recipe * Add news * Update canary-release action version (conda#11799) * Updated conda CLI help to be better for users. (conda#11708) Co-authored-by: Jannis Leidel <jannis@leidel.info> Co-authored-by: Bianca Henderson <bhenderson@anaconda.com> * 🔄 Synced file(s) with conda/infra (conda#11827) Co-authored-by: Conda Bot <conda-bot@users.noreply.github.com> * Fix pypy win (conda#11734) * [pre-commit.ci] pre-commit autoupdate (conda#11833) updates: - [github.com/akaihola/darker: 1.5.0 → 1.5.1](akaihola/darker@1.5.0...1.5.1) - [github.com/PyCQA/pylint: v2.15.0 → v2.15.2](pylint-dev/pylint@v2.15.0...v2.15.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fixes an issue that we are having with carnary builds (conda#11803) * Fixes an issue that we are having with carnary builds See this issue for more information: conda/conda-build#4572 * undoing upates to cytoolz * Update CONTRIBUTING.md (conda#11816) Change all references to `triaging` to `issue sorting` per conda/infrastructure#575 * Release 22.9.0 (conda#11844) * Updated .authors.yml * Updated .mailmap * Updated news * Updated authorship for 22.9.0 * Updated CHANGELOG for 22.9.0 * Added first contributions * Update CHANGELOG.md Co-authored-by: Ken Odegard <kodegard@anaconda.com> * Update CHANGELOG.md Co-authored-by: Ken Odegard <kodegard@anaconda.com> * Update CHANGELOG.md Co-authored-by: Ken Odegard <kodegard@anaconda.com> * Update CHANGELOG.md Co-authored-by: Ken Odegard <kodegard@anaconda.com> * Update CHANGELOG.md Co-authored-by: Ken Odegard <kodegard@anaconda.com> * Add special announcement to changelog. * Remove emoji to not freak out rever. Co-authored-by: Jannis Leidel <jannis@leidel.info> Co-authored-by: Ken Odegard <kodegard@anaconda.com> * Expose --mach and --arch to devenv script (conda#11851) * [pre-commit.ci] pre-commit autoupdate (conda#11859) updates: - [github.com/PyCQA/pylint: v2.15.2 → v2.15.3](pylint-dev/pylint@v2.15.2...v2.15.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * 🔄 Synced file(s) with conda/infra (conda#11853) Co-authored-by: Conda Bot <conda-bot@users.noreply.github.com> * Require `ruamel.yaml` (conda#11868) * Update requirements.txt (conda#11842) Updating the flask to address multiple CVEs including ones listed in conda#11831 * Grammatical correction in docs (conda#11870) * Removed allow_other_channels from conda docs. (conda#11866) * Use maintained `ruamel.yaml` (conda#11837) * Try using maintained ruamel.yaml The conda specific fork is maintained as a conda package only in https://github.com/conda-forge/ruamel_yaml-feedstock/tree/master/recipe. My understanding of what's being patched there seems to indicate it's just the __init__.py file. I'm hoping just a rename of the import will be enough, or at least surface what real problems exist. Ref conda#10691 * Only use ruamel.yaml * Use updated ruamel.yaml API stringio * Cache yaml parsers * TEMP CI CHANGE, REMOVE BEFORE MERGE * Add ruamel_yaml fallback * Add news * Undo CI fixes * Update news Co-authored-by: YuviPanda <yuvipanda@gmail.com> Co-authored-by: Orion Poplawski <orion@nwra.com> * document conda-build: pkg_format: 2 (conda#11881) * document conda-build: pkg_format: 2 * Apply suggestions from code review Co-authored-by: Katherine Kinnaman <kkinnaman@anaconda.com> Co-authored-by: Cheng H. Lee <chenghlee@users.noreply.github.com> Co-authored-by: Ken Odegard <kodegard@anaconda.com> Co-authored-by: Katherine Kinnaman <kkinnaman@anaconda.com> Co-authored-by: Jannis Leidel <jannis@leidel.info> Co-authored-by: Conda Bot <18747875+conda-bot@users.noreply.github.com> Co-authored-by: Conda Bot <conda-bot@users.noreply.github.com> Co-authored-by: Isuru Fernando <isuruf@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Travis Hathaway <travis.j.hathaway@gmail.com> Co-authored-by: Shaun Walbridge <46331011+scdub@users.noreply.github.com> Co-authored-by: Nikhil Raverkar <raverkar.nikhil@gmail.com> Co-authored-by: Satyam Vyas <vyas.satyam.04@gmail.com> Co-authored-by: YuviPanda <yuvipanda@gmail.com> Co-authored-by: Orion Poplawski <orion@nwra.com> Co-authored-by: Daniel Holth <dholth@anaconda.com>

beeankha · 2022-09-29T19:06:24Z

Some of the CI test failures are due to this issue: #11905

jezdez

Besides the naming issue, this LGTM!

conda/plugins/__init__.py

conda/core/index.py

jezdez

I think this is as good as it gets. @beeankha @travishathaway @kenodegard Would you mind a final review?

news/11854-virtual-packages-plugins

jezdez

🚀

kenodegard

LGTM 🚀

* Add hook for virtual package plugins * Add documentation on virtual package plugins * Add virtual package plugin hook test * Fix virtual packages unit test, update index module * Add news item * Remove virtual package code from index.py, add plugins and register them in context.py * Update 'available version' on plugins docs page * Update virtual packages plugin hook naming for consistency * Mark conda.common.cuda.cuda_detect for deprecation. * Use functools.wraps for properly wrapping functions. * Be explicit about exception. * Move and fix cuda tests. * Add a cached version to make sure conda.common.cuda.cuda_detect stays uncached. * Update news file with PR number * Move _supplement_index_with_system back up to where it was previous to refactoring * Using for-else and contextlib.suppress in cuda virtual pkg plugin * Update `maxsize` to None in `lru_cache()` decorator for efficiency * Fix virtual packages unit test, update index module * Move virtual package plugins into submodule. * Update structure of virtual package plugins, re-do imports * Fixes issue with tests plus more The order of decorator functions was causing an issue with tests. Before this commit we had the following order: cache -> env_var_override Now it is: env_var_override -> cache This means that the env_var_override will always override the value where before this was being cached. Also changed in this commit is the adding back of the `cuda_version` to the `context` object. I figured this would be a good idea because removing this may break other projects out there that have come to depend on it. * Update test_solve.py file to make tests pass * Add pluggy as a requirement to setup.py and recipe files, update test_create to test for Python3 vs 2 * Clear cache in three more cuda tests, change python version to 3.8 to enable test to pass * Change location of cache clearing in virtual package cuda tests * Mark conda.base.context.cuda_version function as pending deprecation. * Refactor plugins package. * Try using a fixture for cache invalidation. * Remove duplicate item in recipe. * Use correct path in autodoc directives Co-authored-by: Ken Odegard <kodegard@anaconda.com> Co-authored-by: Filipe Lains <lains@riseup.net> Co-authored-by: Jannis Leidel <jannis@leidel.info> Co-authored-by: Travis Hathaway <travis.j.hathaway@gmail.com>

beeankha self-assigned this Sep 16, 2022

beeankha changed the base branch from main to feature/plugins September 16, 2022 20:48

conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Sep 16, 2022

beeankha force-pushed the virtual_pkgs_hook branch from 9185037 to e59cebd Compare September 16, 2022 20:52

beeankha added source::anaconda created by members of Anaconda, Inc. plugins pertains to a plugin/subcommand in-progress issue is actively being worked on labels Sep 16, 2022

beeankha force-pushed the virtual_pkgs_hook branch 4 times, most recently from 739cf4e to b42772f Compare September 21, 2022 15:04

beeankha removed the in-progress issue is actively being worked on label Sep 21, 2022

beeankha marked this pull request as ready for review September 21, 2022 17:52

beeankha requested a review from a team as a code owner September 21, 2022 17:52

kenodegard reviewed Sep 22, 2022

View reviewed changes

beeankha mentioned this pull request Sep 23, 2022

Virtual Package Plugins #11826

Closed

2 tasks

beeankha changed the title ~~Implement Virtual Packages Plugin Hook~~ [WIP] Implement Virtual Packages Plugin Hook Sep 26, 2022

beeankha marked this pull request as draft September 26, 2022 17:36

beeankha force-pushed the virtual_pkgs_hook branch from 0a49f45 to 8f37d52 Compare September 28, 2022 21:58

beeankha changed the title ~~[WIP] Implement Virtual Packages Plugin Hook~~ [WIP] Implement Virtual Packages Plugins Oct 3, 2022

beeankha marked this pull request as ready for review October 4, 2022 02:32

beeankha changed the title ~~[WIP] Implement Virtual Packages Plugins~~ Implement Virtual Packages Plugins Oct 5, 2022

jezdez previously approved these changes Oct 5, 2022

View reviewed changes

conda/plugins/__init__.py Outdated Show resolved Hide resolved

beeankha dismissed jezdez’s stale review via aaedbdf October 5, 2022 13:10

jezdez previously approved these changes Oct 5, 2022

View reviewed changes

conda/core/index.py Outdated Show resolved Hide resolved

jezdez dismissed their stale review via bac854e October 5, 2022 13:19

Fix docs.

e2f8ba5

jezdez changed the base branch from feature/plugins to main October 28, 2022 15:38

jezdez mentioned this pull request Oct 28, 2022

Move get_plugin_manager to conda.plugins #11836

Closed

3 tasks

jezdez and others added 10 commits October 28, 2022 18:56

Merge branch 'main' into virtual_pkgs_hook

683f6b8

Rename folder.

af67d3d

Remove wrong code.

9c648f3

Remove old docs section.

1809bf7

Remove duplicate item in recipe.

bec288d

Merge branch 'main' into virtual_pkgs_hook

d6d44d5

Merge branch 'main' into virtual_pkgs_hook

6eaaac6

Use correct path in autodoc directives

167859c

Remove tilde

38b7e3c

Move docs into docstring.

ea1f37e

jezdez previously approved these changes Nov 1, 2022

View reviewed changes

beeankha commented Nov 1, 2022

View reviewed changes

news/11854-virtual-packages-plugins Outdated Show resolved Hide resolved

Update news/11854-virtual-packages-plugins

73715e4

beeankha dismissed jezdez’s stale review via 73715e4 November 1, 2022 12:57

travishathaway approved these changes Nov 1, 2022

View reviewed changes

travishathaway requested a review from kenodegard November 1, 2022 12:58

Merge branch 'main' into virtual_pkgs_hook

2651937

jezdez approved these changes Nov 1, 2022

View reviewed changes

kenodegard approved these changes Nov 1, 2022

View reviewed changes

beeankha merged commit 7172d59 into conda:main Nov 1, 2022

XuehaiPan mentioned this pull request Nov 30, 2022

Detect CUDA driver version in subprocess #11667

Merged

3 tasks

chrisburr mentioned this pull request Jan 6, 2023

CONDA_SUBDIR doesn't change virtual packages #12219

Closed

2 tasks

This was referenced Feb 15, 2023

Plugin mechanism for virtual packages #10131

Closed

Remove deprecated virtual package-related features in 23.9 #12384

Closed

github-actions bot added the locked [bot] locked due to inactivity label Nov 2, 2023

github-actions bot locked as resolved and limited conversation to collaborators Nov 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement Virtual Packages Plugins #11854

Implement Virtual Packages Plugins #11854

Uh oh!

beeankha commented Sep 16, 2022 •

edited

Loading

Uh oh!

kenodegard left a comment

Uh oh!

jezdez commented Sep 23, 2022

Uh oh!

beeankha commented Sep 29, 2022

Uh oh!

jezdez left a comment

Uh oh!

Uh oh!

Uh oh!

jezdez left a comment

Uh oh!

Uh oh!

jezdez left a comment

Uh oh!

kenodegard left a comment

Uh oh!

Uh oh!

Implement Virtual Packages Plugins #11854

Implement Virtual Packages Plugins #11854

Uh oh!

Conversation

beeankha commented Sep 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kenodegard left a comment

Choose a reason for hiding this comment

Uh oh!

jezdez commented Sep 23, 2022

Uh oh!

beeankha commented Sep 29, 2022

Uh oh!

jezdez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jezdez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jezdez left a comment

Choose a reason for hiding this comment

Uh oh!

kenodegard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

beeankha commented Sep 16, 2022 •

edited

Loading