Skip to content

Conversation

jezdez
Copy link
Member

@jezdez jezdez commented Mar 24, 2022

This starts a longer journey in removing Python 2 related code. I've tried to separate the code into individual commits to it's easier to review.

@anaconda-issue-bot anaconda-issue-bot added the cla-signed [bot] added once the contributor has signed the CLA label Mar 24, 2022
@jezdez
Copy link
Member Author

jezdez commented Mar 24, 2022

pre-commit.ci autofix

@kenodegard kenodegard closed this Mar 24, 2022
@kenodegard kenodegard reopened this Mar 24, 2022
@kenodegard kenodegard added this to the 4.13.0 milestone Mar 24, 2022
@jezdez jezdez changed the title Legacy Python removal WIP Legacy Python removal Mar 31, 2022
@kenodegard kenodegard force-pushed the py2isms-removal branch 2 times, most recently from 131b1d6 to e5cfc47 Compare May 3, 2022 14:24
Copy link
Contributor

@kenodegard kenodegard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jezdez All passing! Given that this PR is over a month old I suggest we merge this in and continue additional work in another legacy removal PR (e.g. removing odict/OrderDict).

@jezdez jezdez changed the title WIP Legacy Python removal Legacy Python removal May 9, 2022
@jezdez jezdez marked this pull request as ready for review May 9, 2022 16:03
@jezdez jezdez requested a review from a team as a code owner May 9, 2022 16:03
@jezdez
Copy link
Member Author

jezdez commented May 9, 2022

@kenodegard You're totally correctly, let's start here and iterate over it.

@jezdez jezdez merged commit dedbb59 into master May 9, 2022
@jezdez jezdez deleted the py2isms-removal branch May 9, 2022 16:21
@tonigi
Copy link

tonigi commented May 27, 2022

I'm under the impression that itervalues is still used by https://github.com/conda/conda-build/blob/1ed8da062f123e1fbca604df26b064b267a3d094/conda_build/utils.py#L2087 . This currently breaks python 3.10 builds (at least for me, on conda-forge with conda 4.13).

huydhn added a commit to pytorch/test-infra that referenced this pull request Apr 14, 2023
… the CI (#4028)

After some debugging, I think I have found the potential root cause of
the flaky missing dependencies on MacOS. It is in the conda clone step
in which only conda packages are copied over but not pip (untracked)
files.

For example, in this failure example
https://github.com/pytorch/pytorch/actions/runs/4692698401/jobs/8318918238
had:

```
Source:      /Users/ec2-user/runner/_work/_temp/conda-python-3.9.12
Destination: /Users/ec2-user/runner/_work/_temp/conda_environment_4692698401
Packages: 58
Files: 0 <--- Nothing here
```

And it used `/Users/ec2-user/runner/_work/_temp/miniconda/bin/conda` as
reported by `which conda`. This conda is correct but old 4.12.0
(https://github.com/pytorch/test-infra/blob/main/.github/actions/setup-miniconda/action.yml#L15).

On the other hand, when working correctly, these files should have been
copied over to the cloned env:

```
Source:      /Users/ec2-user/runner/_work/_temp/conda-python-3.9.12
Destination: /Users/ec2-user/runner/_work/_temp/conda_environment_4683905314
Packages: 58
Files: 13010 <-- These files are copied over
```

This conda here was `/opt/homebrew/Caskroom/miniconda/base/bin/conda` on
M1 and `/usr/local/bin/conda` on x86_64.
Both are newer (23.1.0). It seems that **we unknowingly use the newer
system-wide conda on MacOS**, which is not part of the CI.

So the attempt fix here is to:

* Upgrade MacOS conda to
[23.1.0](https://repo.anaconda.com/miniconda/Miniconda3-py39_23.1.0-1-MacOSX-arm64.sh).
My theory is that the conda clone issue was fixed by this
conda/conda#11364
* Always use `/Users/ec2-user/runner/_work/_temp/miniconda/bin/conda` as
this is supposed to be the one controlled by the CI
* Check if the number of untracked files (pip dependencies) is 0. If it
is, reinstall pip requirements directly on the cloned conda environment

### Testing


https://github.com/pytorch/pytorch/actions/runs/4695636286/jobs/8338451917

* Use `/Users/ec2-user/runner/_work/_temp/miniconda/bin/conda`
* Use `conda 23.1.0`
* Files are copied correctly
@github-actions github-actions bot added the locked [bot] locked due to inactivity label May 27, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla-signed [bot] added once the contributor has signed the CLA locked [bot] locked due to inactivity
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

5 participants