Skip to content

Conversation

maflcko
Copy link
Member

@maflcko maflcko commented Mar 27, 2023

Currently the CI env has many intermittent issues:

  • The Ubuntu package servers are frequently down
  • Occasionally other stuff is down, such as dnf, pip, or the android sdk
  • Installing packages is slower than downloading them, at least on Cirrus, which has a fast download speed

Fix all issues by using the Cirrus CI dockerfile env.

@DrahtBot
Copy link
Contributor

DrahtBot commented Mar 27, 2023

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK josibake

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

No conflicts as of last run.

@DrahtBot DrahtBot added the Tests label Mar 27, 2023
@maflcko maflcko marked this pull request as draft March 27, 2023 09:13
@maflcko maflcko force-pushed the 2303-ci-env- branch 2 times, most recently from ddf4cba to e31f87d Compare March 27, 2023 10:39
@maflcko
Copy link
Member Author

maflcko commented Mar 27, 2023

No idea why CI fails. Seems to be related to https://superuser.com/questions/1642858/git-on-debian-10-backports-throws-fatal-unable-to-access-https-github-com-us

https://cirrus-ci.com/task/5391244998737920:

git is already the newest version (1:2.30.2-1~bpo10+1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
if [ "$CIRRUS_PR" = "" ]; then exit 0; fi
git fetch --depth=1 $CIRRUS_REPO_CLONE_URL "pull/${CIRRUS_PR}/merge"
fatal: unable to access 'https://github.com/bitcoin/bitcoin.git/': Failed sending HTTP2 data

@maflcko maflcko marked this pull request as ready for review March 29, 2023 12:20
@maflcko maflcko force-pushed the 2303-ci-env- branch 6 times, most recently from 0e6363e to 64c118f Compare March 29, 2023 13:28
maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Mar 29, 2023
This is needed to work around bitcoin#27340 (comment)

The only change should be that python3.7 is bumped to 3.8, but this is fine because ci/test/00_setup_env_native_qt5.sh still checks for python3.7 compatibility.
maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Mar 31, 2023
This is needed to work around
bitcoin#27340 (comment)

The only change should be that python3.7 is bumped to 3.8, but this is
fine because ci/test/00_setup_env_native_qt5.sh still checks for
python3.7 compatibility.
maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Mar 31, 2023
This is needed to work around
bitcoin#27340 (comment)

The only change should be that python3.7 is bumped to 3.8, but this is
fine because ci/test/00_setup_env_native_qt5.sh still checks for
python3.7 compatibility.
@maflcko maflcko force-pushed the 2303-ci-env- branch 2 times, most recently from fab6de4 to fa90e47 Compare March 31, 2023 14:26
maflcko pushed a commit to maflcko/bitcoin-core that referenced this pull request Apr 7, 2023
This is needed to work around
bitcoin#27340 (comment)

The only change should be that python3.7 is bumped to 3.8, but this is
fine because ci/test/00_setup_env_native_qt5.sh still checks for
python3.7 compatibility.
@maflcko maflcko requested review from adamjonas and removed request for adamjonas April 10, 2023 09:44
MarcoFalke added 2 commits April 11, 2023 14:12
This should give faster feedback about the CI result, while still
keeping expenses reasonable.
This is needed to work around
bitcoin#27340 (comment)

The only change should be that python3.7 is bumped to 3.8, but this is
fine because ci/test/00_setup_env_native_qt5.sh still checks for
python3.7 compatibility.
Copy link
Member

@josibake josibake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code review ACK fa4a46d

read through the linked documentation from cirrus and this definitely makes sense. I tried to run the ci locally using cirrus run but ran into some issues (unrelated to this PR). If you have any tips on how to test this, happy to give it a go, but the code looks good otherwise.

@@ -47,6 +47,7 @@ container_depends_template: &CONTAINER_DEPENDS_TEMPLATE
cpu: 2
greedy: true
memory: 8G # Set to 8GB to avoid OOM. https://cirrus-ci.org/guide/linux/#linux-containers
dockerfile: ci/test_imagefile # https://cirrus-ci.org/guide/docker-builder-vm/#dockerfile-as-a-ci-environment
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note to self/reviewers: this file was introduced in #26976

@@ -195,6 +197,7 @@ task:
image: debian:bullseye
cpu: 2
memory: 8G
# docker_arguments: # Can use dockerfile after https://github.com/cirruslabs/cirrus-ci-docs/issues/1154
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I grep for "TODO"s a lot when looking for stuff to work on, might be nice to add a todo comment here to make it easier for others to find later on

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not really something to work on, other than maybe subscribing to the linked issue and follow-up when it is closed?

@@ -210,6 +210,7 @@ task:
docker_arguments:
CI_IMAGE_NAME_TAG: ubuntu:jammy
FILE_ENV: "./ci/test/00_setup_env_win64.sh"
<< : *CREDITS_TEMPLATE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is unrelated to switching to the docker file, right? would prefer this change in it's own PR because I have a few questions about cost (before / after, etc), but not a super big deal

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to answer any questions. The cost will be displayed on every task in a badge.

@maflcko
Copy link
Member Author

maflcko commented Apr 14, 2023

I tried to run the ci locally using cirrus run but ran into some issues (unrelated to this PR). If you have any tips on how to test this

I can help with running the CI locally, but not how to run Cirrus locally. I don't think testing this is possible, other than looking at the CI tasks for this pull?

@josibake
Copy link
Member

I can help with running the CI locally, but not how to run Cirrus locally. I don't think testing this is possible, other than looking at the CI tasks for this pull?

I installed the cirrus cli tool and ran cirrus run in the directory, which then picks up the .cirrus.yml file and executes with cirrus workers, but was running into a bunch of permissions issues. Not related to this PR, so I'm not going to try and troubleshoot it further

@maflcko maflcko requested a review from fanquake April 17, 2023 12:24
@fanquake
Copy link
Member

Going to merge this now, so that 25.x and master will have the same CI infra.

@fanquake fanquake merged commit 5165984 into bitcoin:master Apr 18, 2023
@maflcko maflcko deleted the 2303-ci-env-🏺 branch April 18, 2023 10:38
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 18, 2023
fanquake added a commit to bitcoin-core/gui that referenced this pull request Apr 28, 2023
fac395e ci: Bump ci/lint/Dockerfile (MarcoFalke)
fa6eb65 test: Use python3.8 pow() (MarcoFalke)
88881cf Bump python minimum version to 3.8 (MarcoFalke)

Pull request description:

  There is no pressing reason to drop support for 3.7, however there are several maintenance issues:

  * There is no supported operating system that ships 3.7 by default. (debian:buster is EOL and unmaintained to the extent that it doesn't run in the CI environment. See bitcoin/bitcoin#27340 (comment))
  * Compiling python 3.7 from source is also unsupported on at least macos, according to bitcoin/bitcoin#24017 (comment)
  * Recent versions of lief require 3.8, see bitcoin/bitcoin#27507 (comment)

  Fix all maintenance issues by bumping the minimum.

ACKs for top commit:
  RandyMcMillan:
    ACK fac395e
  fjahr:
    ACK fac395e
  fanquake:
    ACK fac395e

Tree-SHA512: c198decdbbe29d186d73ea3f6549d8a38479383495d14a965a2f9211ce39637b43f13a4c2a5d3bf56e2d468be4bbe49b4ee8e8e19ec69936ff43ddf2b714c712
RandyMcMillan pushed a commit to RandyMcMillan/bitcoin that referenced this pull request May 27, 2023
This is needed to work around
bitcoin#27340 (comment)

The only change should be that python3.7 is bumped to 3.8, but this is
fine because ci/test/00_setup_env_native_qt5.sh still checks for
python3.7 compatibility.
janus pushed a commit to BitgesellOfficial/bitgesell that referenced this pull request Sep 3, 2023
This is needed to work around
bitcoin/bitcoin#27340 (comment)

The only change should be that python3.7 is bumped to 3.8, but this is
fine because ci/test/00_setup_env_native_qt5.sh still checks for
python3.7 compatibility.
knst pushed a commit to knst/dash that referenced this pull request Oct 19, 2023
fac395e ci: Bump ci/lint/Dockerfile (MarcoFalke)
fa6eb65 test: Use python3.8 pow() (MarcoFalke)
88881cf Bump python minimum version to 3.8 (MarcoFalke)

Pull request description:

  There is no pressing reason to drop support for 3.7, however there are several maintenance issues:

  * There is no supported operating system that ships 3.7 by default. (debian:buster is EOL and unmaintained to the extent that it doesn't run in the CI environment. See bitcoin#27340 (comment))
  * Compiling python 3.7 from source is also unsupported on at least macos, according to bitcoin#24017 (comment)
  * Recent versions of lief require 3.8, see bitcoin#27507 (comment)

  Fix all maintenance issues by bumping the minimum.

ACKs for top commit:
  RandyMcMillan:
    ACK fac395e
  fjahr:
    ACK fac395e
  fanquake:
    ACK fac395e

Tree-SHA512: c198decdbbe29d186d73ea3f6549d8a38479383495d14a965a2f9211ce39637b43f13a4c2a5d3bf56e2d468be4bbe49b4ee8e8e19ec69936ff43ddf2b714c712
knst pushed a commit to knst/dash that referenced this pull request Oct 19, 2023
fac395e ci: Bump ci/lint/Dockerfile (MarcoFalke)
fa6eb65 test: Use python3.8 pow() (MarcoFalke)
88881cf Bump python minimum version to 3.8 (MarcoFalke)

Pull request description:

  There is no pressing reason to drop support for 3.7, however there are several maintenance issues:

  * There is no supported operating system that ships 3.7 by default. (debian:buster is EOL and unmaintained to the extent that it doesn't run in the CI environment. See bitcoin#27340 (comment))
  * Compiling python 3.7 from source is also unsupported on at least macos, according to bitcoin#24017 (comment)
  * Recent versions of lief require 3.8, see bitcoin#27507 (comment)

  Fix all maintenance issues by bumping the minimum.

ACKs for top commit:
  RandyMcMillan:
    ACK fac395e
  fjahr:
    ACK fac395e
  fanquake:
    ACK fac395e

Tree-SHA512: c198decdbbe29d186d73ea3f6549d8a38479383495d14a965a2f9211ce39637b43f13a4c2a5d3bf56e2d468be4bbe49b4ee8e8e19ec69936ff43ddf2b714c712
PastaPastaPasta pushed a commit to knst/dash that referenced this pull request Oct 23, 2023
fac395e ci: Bump ci/lint/Dockerfile (MarcoFalke)
fa6eb65 test: Use python3.8 pow() (MarcoFalke)
88881cf Bump python minimum version to 3.8 (MarcoFalke)

Pull request description:

  There is no pressing reason to drop support for 3.7, however there are several maintenance issues:

  * There is no supported operating system that ships 3.7 by default. (debian:buster is EOL and unmaintained to the extent that it doesn't run in the CI environment. See bitcoin#27340 (comment))
  * Compiling python 3.7 from source is also unsupported on at least macos, according to bitcoin#24017 (comment)
  * Recent versions of lief require 3.8, see bitcoin#27507 (comment)

  Fix all maintenance issues by bumping the minimum.

ACKs for top commit:
  RandyMcMillan:
    ACK fac395e
  fjahr:
    ACK fac395e
  fanquake:
    ACK fac395e

Tree-SHA512: c198decdbbe29d186d73ea3f6549d8a38479383495d14a965a2f9211ce39637b43f13a4c2a5d3bf56e2d468be4bbe49b4ee8e8e19ec69936ff43ddf2b714c712
backpacker69 pushed a commit to peercoin/peercoin that referenced this pull request Mar 5, 2024
This is needed to work around
bitcoin/bitcoin#27340 (comment)

The only change should be that python3.7 is bumped to 3.8, but this is
fine because ci/test/00_setup_env_native_qt5.sh still checks for
python3.7 compatibility.
@bitcoin bitcoin locked and limited conversation to collaborators Apr 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants