ci: Upgrade most ci configs to focal #19267

maflcko · 2020-06-13T13:12:01Z

Generally developers compile with recent compilers, so bumping the ci configs to a recent OS should be uncontroversial. Older OSes (especially with compiler sanitizers) need workarounds that can be dropped by running on a more recent OS.

This pull changes the asan sanitizer and the experimental multiprocess build to use focal.
Also, it runs the no_wallet config on xenial to test against python 3.5, according to doc/dependencies.md.

Finally, all configs that mimic gitian (win and mac) will stay at bionic.

hebasto · 2020-06-13T13:42:31Z

.travis.yml

@@ -129,13 +115,12 @@ jobs:
        FILE_ENV="./ci/test/00_setup_env_native_fuzz.sh"

    - stage: test
-      name: 'x86_64 Linux [GOAL: install]  [bionic]  [multiprocess]'
-      if: type != pull_request OR commit_message =~ /depends:|multiprocess:/ # Skip on non-depends, non-multiprocess PRs


Why this if: removed?

I think it is not worth the risk of having the build go broken unnoticed as opposed to the cost of needing one more vm. The travis contract is expiring this month and we will either renew it or buy a contract at a different provider.

maflcko · 2020-06-13T14:21:55Z

For some reason this doesn't link:

/usr/bin/ld: qt/libbitcoinqt.a(libbitcoinqt_a-bitcoin.o): in function `GuiMain(int, char**)':

/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/qt/bitcoin.cpp:444: undefined reference to `qInitResources_bitcoin()'

/usr/bin/ld: /home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/qt/bitcoin.cpp:445: undefined reference to `qInitResources_bitcoin_locale()'

clang: error: linker command failed with exit code 1 (use -v to see invocation)

locally everything works fine, but on travis not: https://travis-ci.org/github/bitcoin/bitcoin/jobs/697962112#L4091

hebasto · 2020-06-13T14:42:55Z

For some reason this doesn't link:

/usr/bin/ld: qt/libbitcoinqt.a(libbitcoinqt_a-bitcoin.o): in function `GuiMain(int, char**)':

/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/qt/bitcoin.cpp:444: undefined reference to `qInitResources_bitcoin()'

/usr/bin/ld: /home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/qt/bitcoin.cpp:445: undefined reference to `qInitResources_bitcoin_locale()'

clang: error: linker command failed with exit code 1 (use -v to see invocation)

locally everything works fine, but on travis not: https://travis-ci.org/github/bitcoin/bitcoin/jobs/697962112#L4091

Could you try to add #include <QDir> into bitcoin.cpp?

DrahtBot · 2020-06-13T14:48:20Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

ci: Run asan ci config on cirrus #19321 (ci: Run asan ci config on cirrus by MarcoFalke)
ci: Run ci configs on cirrus #19179 ([WIP RFC DONOTMERGE] ci: Run ci configs on cirrus by MarcoFalke)
wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets #19077 (wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets by achow101)
ci: Run fuzz testing test cases (bitcoin-core/qa-assets) under valgrind to catch memory errors #18912 (ci: Run fuzz testing test cases (bitcoin-core/qa-assets) under valgrind to catch memory errors by practicalswift)
build: Allow BDB between 4.8 and 5.3 without --with-incompatible-bdb #18870 (build: Allow BDB between 4.8 and 5.3 without --with-incompatible-bdb by achow101)
net: Add NAT-PMP port forwarding support #18077 (net: Add NAT-PMP port forwarding support by hebasto)
External signer support - Wallet Box edition #16546 (External signer support - Wallet Box edition by Sjors)
util: add RunCommandParseJSON #15382 (util: add runCommandParseJSON by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

kiminuo · 2020-06-13T15:52:05Z

(@hebasto Different PR but similar error. However, your suggestion did not help me: https://travis-ci.com/github/kiminuo/bitcoin/jobs/348750106 https://travis-ci.com/github/kiminuo/bitcoin/jobs/348750106#L4065)

hebasto · 2020-06-13T16:46:58Z

@MarcoFalke
It seems the root is here:
https://travis-ci.org/github/bitcoin/bitcoin/jobs/697962112#L3568
and
https://travis-ci.org/github/bitcoin/bitcoin/jobs/697962112#L3928

maflcko · 2020-06-13T16:59:11Z

Nice catch. Though,

Why does this error not abort the build immediately?
Why does it not reproduce locally?

hebasto · 2020-06-13T17:51:23Z

Travis is happy with 17dc996 :)

fanquake · 2020-06-14T01:16:04Z

Concept ACK.

Generally developers compile with recent compilers, so bumping the ci configs to a recent OS should be uncontroversial.

If we're going to do this I think we should make a decision in #18921. Otherwise the way we compile and test in our CI (as well as the way some developers are compiling locally) is starting to diverge from the way we actually build releases.

For 0.21 this will be Ubuntu Bionic with python3.6.

Is the minimum supported Python being bumped to 3.6 for 0.21, or are we just not going to test Python 3.5 compat any more? You said here:

https://packages.ubuntu.com/xenial/python3 is still supported and not EOL until April next year, so we can't bump to python 3.6 yet.

Also, bump the travis env to bionic. This shouldn't matter at all because all ci configs run inside a docker, but it does seem to fix a bug. See bitcoin#19267 (comment)

maflcko · 2020-06-14T12:14:40Z

starting to diverge from the way we actually build releases.

I don't mind bumping gitian to focal, but I think that discussion should happen elsewhere. I kept the three gitian-mimic configs at bionic (win64, mac, and the C++17 linux build). Though, sanitizers are not supported by gitian, so I think requiring them to use the older LTS and patch them with workarounds might not be the best way forward.

Is the minimum supported Python being bumped to 3.6 for 0.21, or are we just not going to test Python 3.5 compat any more?

I am testing python 3.5 compat, but most of my testing has moved out of this repo. Though, I've switched the existing no_wallet config to xenial in the last commit to make it a bit more explicit that python3.5 compat is not changed by this pull.

Also, bump the travis env to bionic. This shouldn't matter at all because all ci configs run inside a docker, but it does seem to fix a bug. See bitcoin#19267 (comment)

practicalswift · 2020-06-14T16:57:21Z

Concept ACK

Also, bump the travis env to bionic. This shouldn't matter at all because all ci configs run inside a docker, but it does seem to fix a bug. See bitcoin#19267 (comment)

maflcko · 2020-06-15T11:56:28Z

Force pushed to get rid of the already merged commit

laanwj · 2020-06-17T14:03:23Z

Concept ACK. I think focal is a better reflection of the compiler and binutils most people are using than bionic, at this point.

Though I agree with @fanquake's point that we're still building releases using bionic, so we need to test that.

However, there should be at least one config to test compile and run-time compatibility on the oldest supported OS. For 0.21 this will be Ubuntu Bionic with python3.6.

This is somewhat confusing to me—compile-time and run-time compatibility was never the same. Run-time compatibility (at least for pre-built releases) historically at least tends to back farther than we support building on. But I suppose backward-testing gitian releases isn't part of the CI so it doesn't matter here.

Also, bump the travis env to bionic. This shouldn't matter at all because all ci configs run inside a docker, but it does seem to fix a bug. See bitcoin#19267 (comment)

maflcko · 2020-06-19T14:34:52Z

doc/dependencies.md says it is python 3.5, so I went ahead and updated OP

Also, bump the travis env to bionic. This shouldn't matter at all because all ci configs run inside a docker, but it does seem to fix a bug. See bitcoin#19267 (comment)

maflcko · 2020-06-19T14:45:23Z

re-arranged commits to minimize diff

Sjors · 2020-06-19T15:07:14Z

ACK fa05f44, assuming Travis passes
(it hasn't shown up in the checks yet)

maflcko · 2020-06-19T16:15:29Z

(it hasn't shown up in the checks yet)

It shows for me:

hebasto · 2020-06-19T16:21:51Z

.travis.yml

-      name: 'x86_64 Linux  [GOAL: install]  [bionic]  [no wallet]'
+      name: 'x86_64 Linux  [GOAL: install]  [xenial]  [no wallet]'


Why xenial?

Please refer to the commit message and pull request description for the motivation behind changes

hebasto

ACK fa05f44

Sjors · 2020-06-19T16:57:24Z

I show up for me now as well.

Also, bump the travis env to bionic. This shouldn't matter at all because all ci configs run inside a docker, but it does seem to fix a bug. See bitcoin#19267 (comment)

Also, bump the travis env to bionic. This shouldn't matter at all because all ci configs run inside a docker, but it does seem to fix a bug. See bitcoin/bitcoin#19267 (comment)

maflcko added the Tests label Jun 13, 2020

maflcko force-pushed the 2006-ciFocal branch from fa27c48 to fa06278 Compare June 13, 2020 13:28

hebasto reviewed Jun 13, 2020

View reviewed changes

kiminuo added a commit to kiminuo/bitcoin that referenced this pull request Jun 13, 2020

https://github.com/bitcoin/bitcoin/pull/19267#issuecomment-643632850

7d70344

DrahtBot mentioned this pull request Jun 13, 2020

ci: Run ci configs on cirrus #19179

Merged

maflcko force-pushed the 2006-ciFocal branch from fa7deb8 to 17dc996 Compare June 13, 2020 16:41

kiminuo added a commit to kiminuo/bitcoin that referenced this pull request Jun 13, 2020

https://github.com/bitcoin/bitcoin/pull/19267#issuecomment-643656137

0ef40ea

This was referenced Jun 13, 2020

ci: Run fuzz testing test cases (bitcoin-core/qa-assets) under valgrind to catch memory errors #18912

Merged

build: Allow BDB between 4.8 and 5.3 without --with-incompatible-bdb #18870

Closed

net: Add NAT-PMP port forwarding support #18077

Merged

maflcko force-pushed the 2006-ciFocal branch from 17dc996 to fa8f771 Compare June 14, 2020 10:42

maflcko force-pushed the 2006-ciFocal branch from fa8f771 to fa654ee Compare June 14, 2020 10:43

maflcko force-pushed the 2006-ciFocal branch from fa654ee to fa71186 Compare June 14, 2020 11:26

maflcko force-pushed the 2006-ciFocal branch from fa71186 to d8e83f8 Compare June 14, 2020 16:45

maflcko force-pushed the 2006-ciFocal branch from d8e83f8 to fa650d9 Compare June 15, 2020 10:30

DrahtBot mentioned this pull request Jun 19, 2020

ci: Run asan ci config on cirrus #19321

Merged

maflcko mentioned this pull request Jun 19, 2020

[WIP DONOTMERGE] Replace boost with C++17 (std::shared_mutex) #19183

Closed

travis: Always run multiprocess build

fa6ddb2

maflcko force-pushed the 2006-ciFocal branch from fa650d9 to faafabb Compare June 19, 2020 14:32

DrahtBot added the Needs rebase label Jun 19, 2020

DrahtBot removed the Needs rebase label Jun 19, 2020

MarcoFalke added 3 commits June 19, 2020 10:43

ci: Have one config run in xenial to test against python3.5

fa88077

Also, bump the travis env to bionic. This shouldn't matter at all because all ci configs run inside a docker, but it does seem to fix a bug. See bitcoin#19267 (comment)

doc: move doc to ci readme

fad6720

ci: Upgrade most ci configs to focal

fa05f44

maflcko force-pushed the 2006-ciFocal branch from faafabb to fa05f44 Compare June 19, 2020 14:44

hebasto reviewed Jun 19, 2020

View reviewed changes

hebasto approved these changes Jun 19, 2020

View reviewed changes

maflcko merged commit 6dc1b45 into bitcoin:master Jun 19, 2020

maflcko deleted the 2006-ciFocal branch June 19, 2020 16:56

bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022

		name: 'x86_64 Linux [GOAL: install] [bionic] [no wallet]'
		name: 'x86_64 Linux [GOAL: install] [xenial] [no wallet]'

ci: Upgrade most ci configs to focal #19267

ci: Upgrade most ci configs to focal #19267

Uh oh!

Conversation

maflcko commented Jun 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hebasto Jun 13, 2020

Choose a reason for hiding this comment

Uh oh!

maflcko Jun 13, 2020

Choose a reason for hiding this comment

Uh oh!

maflcko commented Jun 13, 2020

Uh oh!

hebasto commented Jun 13, 2020

Uh oh!

DrahtBot commented Jun 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

kiminuo commented Jun 13, 2020

Uh oh!

hebasto commented Jun 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maflcko commented Jun 13, 2020

Uh oh!

hebasto commented Jun 13, 2020

Uh oh!

fanquake commented Jun 14, 2020

Uh oh!

maflcko commented Jun 14, 2020

Uh oh!

practicalswift commented Jun 14, 2020

Uh oh!

maflcko commented Jun 15, 2020

Uh oh!

laanwj commented Jun 17, 2020

Uh oh!

maflcko commented Jun 19, 2020

Uh oh!

maflcko commented Jun 19, 2020

Uh oh!

Sjors commented Jun 19, 2020

Uh oh!

maflcko commented Jun 19, 2020

Uh oh!

hebasto Jun 19, 2020

Choose a reason for hiding this comment

Uh oh!

maflcko Jun 19, 2020

Choose a reason for hiding this comment

Uh oh!

hebasto Jun 19, 2020

Choose a reason for hiding this comment

Uh oh!

hebasto left a comment

Choose a reason for hiding this comment

Uh oh!

Sjors commented Jun 19, 2020

Uh oh!

Uh oh!

maflcko commented Jun 13, 2020 •

edited

Loading

DrahtBot commented Jun 13, 2020 •

edited

Loading

hebasto commented Jun 13, 2020 •

edited

Loading