Skip to content

Conversation

maflcko
Copy link
Member

@maflcko maflcko commented Jun 7, 2023

The Android task has many issues:

  • It runs into more network timeouts (intermittent failures) than other tasks
  • It never failed since its introduction years ago in a scenario where all other tasks passed, thus it is useless (so far)

Fix all issues by removing the task. Note that the CI env file is kept, so anyone can still run the Android CI.

Also, use the compute credits to promote tsan, a more useful task.

@DrahtBot
Copy link
Contributor

DrahtBot commented Jun 7, 2023

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK dergoegge, glozow
Concept NACK jarolrod
Concept ACK fanquake

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

@DrahtBot DrahtBot added the Tests label Jun 7, 2023
@fanquake
Copy link
Member

fanquake commented Jun 7, 2023

Concept ACK

Copy link
Member

@dergoegge dergoegge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK fa22538 - nuke it

I have only ever seen this task pass (even if all others fail) or fail intermittently.

@jarolrod
Copy link
Member

jarolrod commented Jun 7, 2023

It should be noted the primary role of the task is to check that a change doesn’t prevent us from building for android, a platform with 3 billion users and a platform that we care about building for. A manual check will be needed for build system changes to ensure that this compatibility is kept.

@dergoegge

can you show an example to back up this claim: "I have only ever seen this task pass (even if all others fail)..." - The android CI won't pass if we can't build core, if all other tests fail we can't build core (macOS 10.15 ci task for example). So this sounds impossible.

Copy link
Member

@jarolrod jarolrod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NACK

I'm not sure if the PR description counts as "many" issues.

If this task was failing intermittently all the time, why are there no issues posted about these intermittent failures occurring all the time, everywhere? We watch this task often in the QML repo and have never had it fail due to a network timeout.

It's great to speed up a task, and I wish all tasks could run as quickly as possible; that would be really great. But not at the expense of a check on building for a platform we want to support, with a massive user base, more than any other operating system in existence.

The potential to allow Android devices to participate in the network as full nodes is powerful. It's beautiful to enable a person in a part of the world where one only has an old Android phone to benefit from having a self-sovereign view of the network and their coins, to participate in the enforcement of the network rules, and to broadcast transactions in an unstoppable P2P manner directly from their device.

And, the changes to support Android are "surprisingly small and sensible", even in the QML repo as we've added the ability for it to run in the background as a process.

Android Tablet Android Phone
FtO9BGDWYAA-t4F Ft-a8omX0AAa3Go

This feels, in a small way, motivated by the want to remove the GUI by some. There seems to be an idea among a handful of us that all of our issues will be fixed if we rm -rf ./src/qt. This isn't so, and maintenance of core will still involve headaches and work. All it does is remove a way for many to run a node, reducing the population and harming the demographic of those running nodes. I care about non-technical individuals being able to run core. I would hope others share the same sentiment.

@maflcko
Copy link
Member Author

maflcko commented Jun 8, 2023

why are there no issues posted about these intermittent failures

Because no one reported them. But they do exist, if you grep for "Warning: An error occurred while preparing SDK package Android Emulator: Connection reset." For example, https://cirrus-ci.com/task/5766445213155328?logs=build#L2700

It's great to speed up a task, and I wish all tasks could run as quickly as possible;

It only checks that it can build, and doesn't run any tests, so I hope the other tasks remain to run the tests.

This feels, in a small way, motivated by the want to remove the GUI by some.

I think this is a far stretch. I've never encouraged to remove the GUI and would strongly object the removal. What this does is remove the Android task from the CI config, nothing more. The CI env is explicitly kept, to be able to continue running the task outside. And in fact it is already running on a nightly basis on my nightly CI repo: https://cirrus-ci.com/task/5872120266227712 , along with other important to check tasks that have no place in this repo.

Needless to say this is also no attempt to remove Android compatibility. I agree with you that this is an important platform and demographic to support.

This is merely removing a task that is failing more often due to unrelated issues (network) than to indicate a useful result (true red CI). If you disagree it would be helpful to demonstrate or link to one instance where this task has caught or would have caught an issue.

I don't feel too strongly, and I am happy to close this, but I think we should be having a discussion based on real data points and not feelings.

@jarolrod
Copy link
Member

jarolrod commented Jun 8, 2023

@MarcoFalke withdrew NACK, I've misunderstood intentions and outcomes of this PR. It is true that right now outside of checking for build support, this doesn't do anything for this repo.

Copy link
Member

@glozow glozow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK fa22538
Agree with removing this task (which doesn't run any tests) to make room for a more useful task (which does run tests).

@maflcko
Copy link
Member Author

maflcko commented Jun 8, 2023

I think it should be fine to re-add this the next time it finds an issue, or when the (unit) tests are run, or some other reason appears to re-add it.

@maflcko
Copy link
Member Author

maflcko commented Jun 8, 2023

Another failure on a doc change: https://cirrus-ci.com/task/5213373952950272?logs=ci#L1404

  CXX      qt/libbitcoinqt_a-qrc_bitcoin.o
  CXX      qt/libbitcoinqt_a-qrc_bitcoin_locale.o
  AR       libbitcoin_node.a
  AR       libbitcoin_util.a
  AR       libbitcoin_wallet.a
  AR       libbitcoin_zmq.a
  AR       libbitcoin_cli.a
  AR       libbitcoin_common.a
  AR       libbitcoin_consensus.a
  CXXLD    crypto/libbitcoin_crypto_base.la
  CXXLD    crypto/libbitcoin_crypto_arm_shani.la
  CXXLD    libunivalue.la
  CXXLD    leveldb/libleveldb.la
  CXXLD    crc32c/libcrc32c.la
  CXXLD    leveldb/libmemenv.la
  AR       libtest_util.a
  AR       qt/libbitcoinqt.a
  CXXLD    qt/bitcoin-qt
  CXXLD    qt/test/test_bitcoin-qt
make[2]: Leaving directory '/tmp/cirrus-ci-build/src'
make[1]: Leaving directory '/tmp/cirrus-ci-build/src'
Making all in doc/man
make[1]: Entering directory '/tmp/cirrus-ci-build/doc/man'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/tmp/cirrus-ci-build/doc/man'
make[1]: Entering directory '/tmp/cirrus-ci-build'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory '/tmp/cirrus-ci-build'
+ cd src/qt
+ ANDROID_HOME=/tmp/cirrus-ci-build/depends/SDKs/android
+ ANDROID_NDK_HOME=/tmp/cirrus-ci-build/depends/SDKs/android/ndk/23.2.8568313
+ make apk
make -C .. bitcoin_qt_apk
make[1]: Entering directory '/tmp/cirrus-ci-build/src'
tar: --exclude=*/*: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tar: --exclude=*/*: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
mkdir -p qt/android/libs/arm64-v8a
cp /tmp/cirrus-ci-build/depends/SDKs/android/ndk/23.2.8568313/toolchains/llvm/prebuilt/linux-x86_64/bin//../sysroot/usr/lib/aarch64-linux-android/libc++_shared.so qt/android/libs/arm64-v8a
tar xf  -C qt/android/src/ src/android/jar/src --strip-components=5
tar: -C: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
make[1]: *** [Makefile:22313: bitcoin_qt_apk] Error 2
make[1]: Leaving directory '/tmp/cirrus-ci-build/src'
make: *** [Makefile:11: apk] Error 2
Exit status: 2

@dergoegge
Copy link
Member

can you show an example to back up this claim: "I have only ever seen this task pass (even if all others fail)..."

@jarolrod I can't find the PR this happened on and I've only seen that one time but I swear I'm not making this up. The macOS task has other build flags, so it must have been some weird combination of that and tests/lint failing but some builds succeeding.

@fanquake fanquake merged commit 62140b5 into bitcoin:master Jun 12, 2023
@maflcko maflcko deleted the 2306-ci-tsan- branch June 12, 2023 15:50
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jun 12, 2023
fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Oct 2, 2023
This was referenced Oct 2, 2023
fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Oct 2, 2023
fanquake added a commit that referenced this pull request Oct 4, 2023
45a5fcb http: bugfix: track closed connection (stickies-v)
752a456 http: log connection instead of request count (stickies-v)
ae86ada http: refactor: use encapsulated HTTPRequestTracker (stickies-v)
f31899d gui: macOS, make appMenuBar part of the main app window (furszy)
64ffa94 gui: macOS, do not process dock icon actions during shutdown (furszy)
e270f3f depends: fix unusable memory_resource in macos qt build (fanquake)
a668394 build, macos: Fix `qt` package build with new Xcode 15 linker (Hennadii Stepanov)
b3517cb test: Test loading wallets with conflicts without a chain (Andrew Chow)
d63478c wallet: Check last block and conflict height are valid in MarkConflicted (Andrew Chow)
5e51a9c ci: Nuke Android APK task, Use credits for tsan (MarcoFalke)
910c362 test: ensure old fee_estimate.dat not read on restart and flushed (ismaelsadeeq)
37764d3 tx fees, policy: read stale fee estimates with a regtest-only option (ismaelsadeeq)
16bb916 tx fees, policy: do not read estimates of old fee_estimates.dat (ismaelsadeeq)
c4dd598 tx fees, policy: periodically flush fee estimates to fee_estimates.dat (ismaelsadeeq)
c36770c test: wallet, verify migration doesn't crash for an invalid script (furszy)
0d2a33e wallet: disallow migration of invalid or not-watched scripts (furszy)
2c51a07 Do not use std::vector = {} to release memory (Pieter Wuille)

Pull request description:

  Further backports for the `25.x` branch. Currently:
  * #27622
  * #27834
  * #28125
  * #28452
  * #28542
  * #28543
  * #28551
  * #28571
  * bitcoin-core/gui#751

ACKs for top commit:
  hebasto:
    re-ACK 45a5fcb, only #28551 has been backported with since my recent [review](#28487 (review)).
  dergoegge:
    reACK 45a5fcb
  willcl-ark:
    reACK 45a5fcb

Tree-SHA512: 0f5807aa364b7c2a2039fef11d5cd5e168372c3bf5b0e941350fcd92e7db4a1662801b97bb4f68e29788c77d24bbf97385a483c4501ca72d93fa25327d5694fa
fanquake added a commit that referenced this pull request Oct 6, 2023
9077f21 depends: fix unusable memory_resource in macos qt build (fanquake)
dccacf0 build, macos: Fix `qt` package build with new Xcode 15 linker (Hennadii Stepanov)
4359649 ci: Switch to `amd64` container in "ARM" task (Hennadii Stepanov)
805f98b ci: Nuke Android APK task, Use credits for tsan (MarcoFalke)
cb5512d test: ensure old fee_estimate.dat not read on restart and flushed (ismaelsadeeq)
01f8ee4 tx fees, policy: read stale fee estimates with a regtest-only option (ismaelsadeeq)
1c98029 tx fees, policy: do not read estimates of old fee_estimates.dat (ismaelsadeeq)
77979e0 tx fees, policy: periodically flush fee estimates to fee_estimates.dat (ismaelsadeeq)
67b6d99 Do not use std::vector = {} to release memory (Pieter Wuille)
defdc15 ci: Use podman stop over podman kill (MarcoFalke)
7f1357d ci: Use podman for persistent workers (MarcoFalke)
0db69a3 ci: Prune dangling images on RESTART_CI_DOCKER_BEFORE_RUN (MarcoFalke)

Pull request description:

  Backports to the 24.x branch. Currently:
  * #27622
  * #27777
  * #27834
  * #27844
  * #27886
  * #28452
  * #28543
  * #28571

ACKs for top commit:
  stickies-v:
    ACK 9077f21

Tree-SHA512: abaafc9a048b67b494993134fd332457ea52695ec007b963c283f962ec40c3b6b3a7e98407481be55d3271a595088a0281cc84b79dad4f24d260381ea0153076
@bitcoin bitcoin locked and limited conversation to collaborators Jun 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants