Skip to content

Update Dockerfile + ci-images flow to build multi arch images #11560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jun 22, 2022

Conversation

dbast
Copy link
Member

@dbast dbast commented Jun 20, 2022

Description

This is done to have linux/aarch64 conda-ci images, which perform better on Apple M1 machines compared to the linux/amd64 images.

The other images (linux/ppc64le, linux/s390x) we basically get for free and prove that the whole tooling is consistently available on all 4 linux platforms.

Once the multi-arch image is published it can be also used during the ci flow to e.g. run a short smoke test on all the non native platforms via emulation.

Checklist - did you ...

  • Add a file to the news directory (using the template) for the next release's release notes?
  • Add / update necessary tests?
  • Add / update outdated documentation?

@dbast dbast requested a review from a team as a code owner June 20, 2022 16:18
@conda-bot conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Jun 20, 2022
Copy link
Member

@jezdez jezdez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat!

Sidenote: I wish we'd have a Windows Docker setup using the servercore or nanocore image to do the same on Windows :-/

@dbast dbast force-pushed the aarch64 branch 2 times, most recently from c08cfec to f772235 Compare June 21, 2022 09:41
@dbast
Copy link
Member Author

dbast commented Jun 21, 2022

yay, all green :)

Had to remove pre-commit in tests/requirements.txt as not available for s390x... that can potentially be moved back after somebody converts the pre-commit-feedstock in conda-forge to be noarch (that should be possible with the right arguments). pre-commit anyway runs via the pre-commit-ci service with auto-commit and does not rely on the conda-ci containers.

@jezdez Regarding your windows container note: what would you expect from windows containers, just more speed? That should be also possible by 1. not installing miniconda (it is pre-installed on all public osx, linux, windows github action runners, see) and 2. using the cache action with a good cache key, e.g. key: ${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('test\requirements') }}-${{ hashFiles('dev\windows\setup.bat') }}-${{ DAY_OF_WEEK }}

@jezdez
Copy link
Member

jezdez commented Jun 21, 2022

yay, all green :)

Had to remove pre-commit in tests/requirements.txt as not available for s390x... that can potentially be moved back after somebody converts the pre-commit-feedstock in conda-forge to be noarch (that should be possible with the right arguments). pre-commit anyway runs via the pre-commit-ci service with auto-commit and does not rely on the conda-ci containers.

Fascinating, might make sense to open a ticket in the feedstock?

FWIW we're relying on pre-commit being installed in the development environment, so

needs to be updated with a note to install pre-commit until the conda-forge recipe is up-to-date.

@jezdez Regarding your windows container note: what would you expect from windows containers, just more speed? That should be also possible by 1. not installing miniconda (it is pre-installed on all public osx, linux, windows github action runners, see) and 2. using the cache action with a good cache key, e.g. key: ${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('test\requirements') }}-${{ hashFiles('dev\windows\setup.bat') }}-${{ DAY_OF_WEEK }}

Yeah, I've been meaning to look into whether we can use the pre-installed miniconda for conda as well and just worry that we end up in a cyclic situation where the CI environment needs to do something different than what the old miniconda did.

Besides that speed component I'm wondering what we can do to further improve the parity between development environments and the CI environment, to reduce the barriers of entry for contributors. A readily available base image might be worth exploring since it'd get Windows devs set up faster. Just a theory though, not really in scope of this PR :)

@dbast
Copy link
Member Author

dbast commented Jun 21, 2022

dbast added 9 commits June 22, 2022 09:02
This is done to have linux/aarch64 conda-ci images, which perform better
on Apple M1 machines compared to the linux/amd64 images.

The other images (linux/ppc64le, linux/s390x) we basically get for free
and prove that the whole tooling is consistently available on all 4 linux
platforms.

Once the multi-arch image is published it can be also used during the ci
flow to e.g. run a short smoke test on all the non-native platforms via
emulation.
@dbast
Copy link
Member Author

dbast commented Jun 22, 2022

Readded pre-commit and removed s390x from the platform list... s390x can be added later when the pre-commit noarch situation is resolved.

adding arm64/ppc64le already serves the purpose of a better Apple M1 experience and prove of consistency.

@dbast
Copy link
Member Author

dbast commented Jun 22, 2022

all green again. ready for another review / merge. thanks

@jezdez jezdez merged commit 23ba63d into conda:master Jun 22, 2022
@dbast dbast deleted the aarch64 branch June 22, 2022 13:32
@github-actions github-actions bot added the locked [bot] locked due to inactivity label Jun 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla-signed [bot] added once the contributor has signed the CLA locked [bot] locked due to inactivity
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants