depends: some base fixes/changes #7809

theuni · 2016-04-04T23:48:20Z

Fix the annoying "unexpected operator" warning that shows up sometimes.
Fix download fallbacks. Fixes depends: Fallback mechanism may not be working properly #7627.
Make sure to check all necessary checksums for cctools/qt.
Ensure (make a good guess) that the host/build toolchains haven't changed since the previous build. If they have, invalidate the necessary packages.
Add a salt option for adding extra info to 4. This would allow us to (for example) seed gitian packages with dpkg info, in order to invalidate the cache if certain conditions were met.

I'm pushing these all in one PR because most of these changes will cause full cache invalidation, so we may as well try to only do it once.

These add very simple sanity checks to ensure that the build/host toolchains have not changed since the last run. If they have, all ids will change and packages will be rebuilt. For more complicated usage (like parsing dpkg), HOST_ID_SALT/BUILD_ID_SALT may be used to introduce arbitrary data to the ids.

In some cases, failed downloads wouldn't trigger a fallback download attempt. Namely, checksum mismatches.

Checksums were being verified after download, but not again before extraction

maflcko · 2016-04-05T09:14:27Z

So the patch to the fallback logic will fix checksum issues like https://travis-ci.org/bitcoin/bitcoin/jobs/112855470#L831 ?

But does it also fix the issues where the fallback logic itself appeared to fail? (c.f. https://travis-ci.org/bitcoin/bitcoin/jobs/112846086#L4033 )

laanwj · 2016-04-05T13:47:24Z

Concept ACK, nice!

theuni · 2016-04-05T17:17:27Z

Yes, it should fix both. It broadens the retry logic, so that a "try" becomes fetch+validate. If the first try fails, it's retried with the fallback url. In the first example, the download appears to succeed but the checksum doesn't match. Since the download was "successful", it doesn't retry. Now it retries the whole thing with the fallback url. A hacked test (i purposely broke the checksum):

cory@cory-i7:~/dev/bitcoin/depends(depends-updates)$ make download
Checksum missing or mismatched for qrencode source. Forcing re-download.
Fetching qrencode-3.4.4.tar.bz2 from https://fukuchi.org/works/qrencode/
2016-04-05 13:20:14 URL:https://fukuchi.org/works/qrencode//qrencode-3.4.4.tar.bz2 [369136/369136] -> "/home/cory/dev/bitcoin/depends/work/download/qrencode-3.4.4/qrencode-3.4.4.tar.bz2.temp" [1]
/home/cory/dev/bitcoin/depends/work/download/qrencode-3.4.4/qrencode-3.4.4.tar.bz2.temp: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
Fetching qrencode-3.4.4.tar.bz2 from https://bitcoincore.org/depends-sources
2016-04-05 13:20:15 URL:http://dev.bitcoincore.org/depends-sources/qrencode-3.4.4.tar.bz2 [369136/369136] -> "/home/cory/dev/bitcoin/depends/work/download/qrencode-3.4.4/qrencode-3.4.4.tar.bz2.temp" [1]
/home/cory/dev/bitcoin/depends/work/download/qrencode-3.4.4/qrencode-3.4.4.tar.bz2.temp: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
make[1]: *** [/home/cory/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-qrencode-qrencode-3.4.4.tar.bz2.hash] Error 1
make: *** [download-osx] Error 2

For the second, I'm not sure why that was a problem before. As-is, it looks like it should try to re-download. But as a test, I've forced a similar failure, and it works as intended.

cory@cory-i7:~/dev/bitcoin/depends(depends-updates)$ make download
Checksum missing or mismatched for qrencode source. Forcing re-download.
Fetching qrencode-3.4.4.tar.bz2 from https://fukuchi.org/works/qrencode/
OpenSSL: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
Unable to establish SSL connection.
Fetching qrencode-3.4.4.tar.bz2 from https://bitcoincore.org/depends-sources
OpenSSL: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
Unable to establish SSL connection.
make[1]: *** [/home/cory/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-qrencode-qrencode-3.4.4.tar.bz2.hash] Error 4
make: *** [download-osx] Error 2

laanwj · 2016-04-06T12:35:27Z

Tested ACK dc4ec6d with my crosstool-ng ARM build

laanwj · 2016-04-06T12:39:11Z

depends/Makefile

+build_id_string+=$(shell $(build_AR) --version 2>/dev/null)
+build_id_string+=$(shell $(build_CXX) --version 2>/dev/null)
+build_id_string+=$(shell $(build_RANLIB) --version 2>/dev/null)
+build_id_string+=$(shell $(build_STRIP) --version 2>/dev/null)


Do we also want to get the linker info?

That's tricky because we don't use the linker directly. It's not specified in depends, because there's no generic way to handle it. I think in 99% of the cases, if the linker changed, one of the above would've changed as well.

True - we never use ld directly, and we shouldn't, for example it would be quite annoying to figure out what the linker is (and what plugin it is using - turns out it is loading /opt/clang39/lib/LLVMgold.so) in my lto setup testing the clang master branch:

CC="${CLANGPATH}/bin/clang " CXX="${CLANGPATH}/bin/clang++" CFLAGS="-flto -O2 -g" CXXFLAGS="-flto -O2 -g" LDFLAGS="-flto -O2 -fuse-ld=gold" RANLIB="${CLANGPATH}/bin/llvm-ranlib"

So yes just wanted to ask to be sure.

11d9f6b depends: qt/cctools: fix checksum checksum tests (Cory Fields) bb717f4 depends: fix "unexpected operator" error during "make download" (Cory Fields) fe740f1 depends: fix fallback downloads (Cory Fields) dc4ec6d depends: create a hostid and buildid and add option for salts (Cory Fields)

Enable macOS cross-compilation Includes code cherry-picked from the following upstream PRs: - bitcoin/bitcoin#7809 - The `native_cctools` fix. - bitcoin/bitcoin#8210 - The macOS toolchain bump. - bitcoin/bitcoin#9891 - bitcoin/bitcoin#15581 - The `tar` change. - bitcoin/bitcoin#16049 - The `native_cctools` change. Build instructions: - Fetch `MacOSX10.11.sdk` from e.g. https://github.com/phracker/MacOSX-SDKs/releases - Extract it into `depends/SDKs` (creating that folder first) - `HOST=x86_64-apple-darwin11 ./zcutil/build.sh`

theuni added 4 commits April 4, 2016 19:26

depends: fix fallback downloads

fe740f1

In some cases, failed downloads wouldn't trigger a fallback download attempt. Namely, checksum mismatches.

depends: fix "unexpected operator" error during "make download"

bb717f4

depends: qt/cctools: fix checksum checksum tests

11d9f6b

Checksums were being verified after download, but not again before extraction

laanwj added the Build system label Apr 5, 2016

laanwj reviewed Apr 6, 2016
View reviewed changes

laanwj merged commit 11d9f6b into bitcoin:master Apr 15, 2016

schinzelh mentioned this pull request Oct 19, 2017

[WIP] Update build system to Bitcoin 0.13.2 dashpay/dash#1692

Closed

24 tasks

str4d mentioned this pull request Oct 6, 2019

Enable macOS cross-compilation zcash/zcash#4148

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

depends: some base fixes/changes #7809

depends: some base fixes/changes #7809

Uh oh!

theuni commented Apr 4, 2016

Uh oh!

maflcko commented Apr 5, 2016

Uh oh!

laanwj commented Apr 5, 2016

Uh oh!

theuni commented Apr 5, 2016

Uh oh!

laanwj commented Apr 6, 2016

Uh oh!

laanwj Apr 6, 2016

Uh oh!

theuni Apr 8, 2016

Uh oh!

laanwj Apr 15, 2016 •

edited

Loading

Uh oh!

Uh oh!

depends: some base fixes/changes #7809

depends: some base fixes/changes #7809

Uh oh!

Conversation

theuni commented Apr 4, 2016

Uh oh!

maflcko commented Apr 5, 2016

Uh oh!

laanwj commented Apr 5, 2016

Uh oh!

theuni commented Apr 5, 2016

Uh oh!

laanwj commented Apr 6, 2016

Uh oh!

laanwj Apr 6, 2016

Choose a reason for hiding this comment

Uh oh!

theuni Apr 8, 2016

Choose a reason for hiding this comment

Uh oh!

laanwj Apr 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

laanwj Apr 15, 2016 •

edited

Loading