depends: switch to secure download of all dependencies #16049

Kemu · 2019-05-19T20:25:54Z

Even if we potentially check the integrity of the downloaded file via hash comparison, we should make use of SSL since it is available.

fanquake · 2019-05-20T04:43:37Z

Concept ACK

If we're going to do this, might as well make the change in all packages.

Looks like http: -> https: will work for everything except miniupnpc. That package could be changed to use the https://miniupnp.tuxfamily.org/files/ mirror.

practicalswift · 2019-05-20T07:33:50Z

Concept ACK

Not leaking is better than leaking.

kristapsk · 2019-05-20T08:25:00Z

Concept ACK

There is no downside in using HTTPS.

jonasschnelli · 2019-05-20T09:18:04Z

Tested ACK (manually downloaded curl -OL https://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz and verified the sha256 hash).

jonatack · 2019-05-20T09:47:20Z

ACK

$ curl -OL https://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz

$ sha256sum db-4.8.30.NC.tar.gz
12edc0df75bf9abd7f82f821795bcee50f42cb2e5f76a6a281b85732798364ef  db-4.8.30.NC.tar.gz

Same hash as the non-SSL http curl from a couple months ago.

jonasschnelli · 2019-05-20T10:09:37Z

@Kemu
Are you interested to also cover the other plain http packages in this pull?

grep -rnw './depends' -e 'http:'
./depends/packages/xtrans.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/lib/
./depends/packages/libxcb.mk:3:$(package)_download_path=http://xcb.freedesktop.org/dist
./depends/packages/freetype.mk:3:$(package)_download_path=http://download.savannah.gnu.org/releases/$(package)
./depends/packages/xextproto.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/proto
./depends/packages/libX11.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/lib/
./depends/packages/native_cctools.mk:8:$(package)_clang_download_path=http://llvm.org/releases/$($(package)_clang_version)
./depends/packages/zlib.mk:3:$(package)_download_path=http://www.zlib.net
./depends/packages/native_cdrkit.mk:3:$(package)_download_path=http://distro.ibiblio.org/fatdog/source/600/c
./depends/packages/xproto.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/proto
./depends/packages/xcb_proto.mk:3:$(package)_download_path=http://xcb.freedesktop.org/dist
./depends/packages/miniupnpc.mk:3:$(package)_download_path=http://miniupnp.free.fr/files
./depends/packages/fontconfig.mk:3:$(package)_download_path=http://www.freedesktop.org/software/fontconfig/release/
./depends/packages/libXau.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/lib/
./depends/packages/bdb.mk:3:$(package)_download_path=http://download.oracle.com/berkeley-db
./depends/packages/libXext.mk:3:$(package)_download_path=http://xorg.freedesktop.org/releases/individual/lib/

laanwj · 2019-05-20T14:43:14Z

utACK, as issues with apt have shown, if in doubt it's better to use TLS, it's one extra layer of security.

maflcko · 2019-05-20T21:31:17Z

Please squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

dongcarl · 2019-05-20T22:06:11Z

Could you add something to depends/packages.md so that future depends packages will be written with this in mind?

Some dependency sources were downloaded via http, even though https (SSL/TLS) options are available. Even if we potentially check the integrity of the downloaded files via hash comparison, we should make use of this additional security layer. bdb.mk fontconfig.mk freetype.mk libX11.mk libXau.mk libXext.mk libxcb.mk native_cctools.mk native_cdrkit.mk xcb_proto.mk xextproto.mk xproto.mk xtrans.mk zlib.mk miniupnp was switched to official project mirror with SSL support

Kemu · 2019-05-20T22:37:07Z

Could you add something to depends/packages.md so that future depends packages will be written with this in mind?

good idea.

  $(package)_download_path:
   Location of the upstream source, without the file-name. Usually http, https
   or ftp. Secure transmission options like https should be preferred if
   available.

Kemu · 2019-05-20T22:54:57Z

make download seems to run without issues, except for native_cdrkit:

Fetching cdrkit-1.1.11.tar.bz2 from https://distro.ibiblio.org/fatdog/source/600/c
...
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

~~while directly running:~~
~~wget https://distro.ibiblio.org/fatdog/source/600/c/cdrkit-1.1.11.tar.bz2~~
or
~~curl https://distro.ibiblio.org/fatdog/source/600/c/cdrkit-1.1.11.tar.bz2 --output cdrkit-1.1.11.tar.bz2~~
~~seem to work just fine.~~

~~Can anyone confirm this behavior in their environment?~~

~~This seems to be a randomly occurring error with curl (only happens with ibiblio.org though):~~
Looks to me like a server configuration issue at ibiblio.org. Probably a loadbalancer with one of the application servers behind it not containing the full cert chain:

**When not working:**
uli@us02:~$ curl -vs https://distro.ibiblio.org
* Rebuilt URL to: https://distro.ibiblio.org/
*   Trying 152.19.134.43...
* TCP_NODELAY set
* Connected to distro.ibiblio.org (152.19.134.43) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* stopped the pause stream!
* Closing connection 0
---
uli@us02:~$ echo | openssl s_client -connect distro.ibiblio.org:443
CONNECTED(00000003)
depth=0 C = US, postalCode = 27514, ST = NC, L = Chapel HIll, street = 153A Country Club Road, O = University of North Carolina at Chapel Hill, CN = distro.ibiblio.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, postalCode = 27514, ST = NC, L = Chapel HIll, street = 153A Country Club Road, O = University of North Carolina at Chapel Hill, CN = distro.ibiblio.org
verify error:num=21:unable to verify the first certificate
verify return:1
---

**When working:**
uli@us02:~$ curl -vs https://distro.ibiblio.org
* Rebuilt URL to: https://distro.ibiblio.org/
*   Trying 152.19.134.43...
* TCP_NODELAY set
* Connected to distro.ibiblio.org (152.19.134.43) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; postalCode=27514; ST=NC; L=Chapel HIll; street=153A Country Club Road; O=University of North Carolina at Chapel Hill; CN=distro.ibiblio.org
*  start date: Feb  1 00:00:00 2017 GMT
*  expire date: Feb  1 23:59:59 2020 GMT
*  subjectAltName: host "distro.ibiblio.org" matched cert's "distro.ibiblio.org"
*  issuer: C=US; ST=MI; L=Ann Arbor; O=Internet2; OU=InCommon; CN=InCommon RSA Server CA
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: distro.ibiblio.org
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html
< Content-Length: 19655
< Date: Mon, 20 May 2019 23:14:43 GMT
< Server: lighttpd/1.4.53
---
uli@us02:~$ echo | openssl s_client -connect distro.ibiblio.org:443
CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA
verify return:1
depth=0 C = US, postalCode = 27514, ST = NC, L = Chapel HIll, street = 153A Country Club Road, O = University of North Carolina at Chapel Hill, CN = distro.ibiblio.org
verify return:1
---

The depends Makefile includes a fallback download path (https://bitcoincore.org/depends-sources), which is probably why the tests are not failing in any case.

DrahtBot · 2019-05-21T01:25:33Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#15844 (depends: Purge libtool archives by dongcarl)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

jonasschnelli · 2019-05-21T09:01:08Z

Thanks.
utACK d8bc47f

practicalswift · 2019-05-21T14:52:43Z

Welcome as a contributor @Kemu! :-)

utACK d8bc47f

dongcarl · 2019-05-21T16:28:54Z

tACK d8bc47f

DrahtBot · 2019-05-22T00:43:31Z

Gitian builds for commit 2d1583e (master):

f55b412069904711bd300117d7bdc212... bitcoin-0.18.99-osx-unsigned.dmg
c21869734ae349430e2d9b68d3c0278d... bitcoin-0.18.99-osx64.tar.gz
352aa189594c0f5d2357c6e4f28bb966... bitcoin-0.18.99-win64-debug.zip
0b6ed0d02ba5133bd99cfc79d5d413ce... bitcoin-0.18.99-win64-setup-unsigned.exe
14a6165ab0f2594aadceea8717c11abc... bitcoin-0.18.99-win64.zip
83b4227efbc50aca017907355a81c6f0... bitcoin-core-osx-0.19-res.yml
8846699e3ebd9686a22f035d6d11060b... bitcoin-core-win-0.19-res.yml
6ed8f8e851a0b9e8a848f40c97925f93... bitcoin-linux-build.log
af43fdb4c90ff38c3996fa15c4eae337... bitcoin-osx-build.log
e6496d658405f10cf9897a78dd58d593... bitcoin-win-build.log

Gitian builds for commit e34ec54 (master and this pull):

0501e05ce75cd4e27a88c7d109d76471... bitcoin-0.18.99-osx-unsigned.dmg
185dc51949c408e96d1d81b9106efcb6... bitcoin-0.18.99-osx64.tar.gz
c3f594ca2dc89ec9b740014ea10f53b5... bitcoin-0.18.99-win64-debug.zip
f96624787ab80df6ad9eca83ec4d5304... bitcoin-0.18.99-win64-setup-unsigned.exe
a2409c2db378a79de7af0f22d5e09af9... bitcoin-0.18.99-win64.zip
4f0a25de54596cceec798c0077929c0c... bitcoin-core-osx-0.19-res.yml
0c61b9d07ed58343057550bdd0b57ecb... bitcoin-core-win-0.19-res.yml
35ee77e88ce21e2fc5980d3dc7f1fe9a... bitcoin-linux-build.log
0722bc2a58e6a296f919ef9de9cb0937... bitcoin-osx-build.log
e732c97c71be1adf5efb807ce0050214... bitcoin-win-build.log

d8bc47f depends: switch to secure download of all dependencies (Ulrich Kempken) Pull request description: Even if we potentially check the integrity of the downloaded file via hash comparison, we should make use of SSL since it is available. ACKs for commit d8bc47: jonasschnelli: utACK d8bc47f practicalswift: utACK d8bc47f dongcarl: tACK d8bc47f Tree-SHA512: e47702f6d243ed7f498ca84c193244382f16f08df6a297caa224b4468f501f3da6fe542fcf3a0dd9c24ab1b0b38bbc51478068e6006a92854ded23abf90de3c8

Enable macOS cross-compilation Includes code cherry-picked from the following upstream PRs: - bitcoin/bitcoin#7809 - The `native_cctools` fix. - bitcoin/bitcoin#8210 - The macOS toolchain bump. - bitcoin/bitcoin#9891 - bitcoin/bitcoin#15581 - The `tar` change. - bitcoin/bitcoin#16049 - The `native_cctools` change. Build instructions: - Fetch `MacOSX10.11.sdk` from e.g. https://github.com/phracker/MacOSX-SDKs/releases - Extract it into `depends/SDKs` (creating that folder first) - `HOST=x86_64-apple-darwin11 ./zcutil/build.sh`

Summary: ``` Some dependency sources were downloaded via http, even though https (SSL/TLS) options are available. Even if we potentially check the integrity of the downloaded files via hash comparison, we should make use of this additional security layer. ``` Backport of core [[bitcoin/bitcoin#16049 | PR16049]]. Depends on D5504. Test Plan: Run the Gitian build twice, check the result is deterministic. Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D5513

…encies d8bc47f depends: switch to secure download of all dependencies (Ulrich Kempken) Pull request description: Even if we potentially check the integrity of the downloaded file via hash comparison, we should make use of SSL since it is available. ACKs for commit d8bc47: jonasschnelli: utACK d8bc47f practicalswift: utACK d8bc47f dongcarl: tACK d8bc47f Tree-SHA512: e47702f6d243ed7f498ca84c193244382f16f08df6a297caa224b4468f501f3da6fe542fcf3a0dd9c24ab1b0b38bbc51478068e6006a92854ded23abf90de3c8

fanquake added the Build system label May 19, 2019

Kemu force-pushed the Berkeley_DB_SSL_source branch from 4a63f1d to 1b7d43e Compare May 20, 2019 21:38

Kemu force-pushed the Berkeley_DB_SSL_source branch from 1b7d43e to d8bc47f Compare May 20, 2019 22:29

Kemu changed the title ~~depends:Enable SSL download of Berkeley DB source~~ depends: switch to secure download of all dependencies May 20, 2019

maflcko added the Needs gitian build label May 21, 2019

DrahtBot removed the Needs gitian build label May 22, 2019

maflcko merged commit d8bc47f into bitcoin:master May 22, 2019

str4d mentioned this pull request Oct 6, 2019

Enable macOS cross-compilation zcash/zcash#4148

Merged

barton2526 mentioned this pull request Sep 13, 2021

build: Misc upstream depends fixes gridcoin-community/Gridcoin-Research#2331

Merged

bitcoin locked as resolved and limited conversation to collaborators Dec 16, 2021

depends: switch to secure download of all dependencies #16049

depends: switch to secure download of all dependencies #16049

Uh oh!

Conversation

Kemu commented May 19, 2019

Uh oh!

fanquake commented May 20, 2019

Uh oh!

practicalswift commented May 20, 2019

Uh oh!

kristapsk commented May 20, 2019

Uh oh!

jonasschnelli commented May 20, 2019

Uh oh!

jonatack commented May 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonasschnelli commented May 20, 2019

Uh oh!

laanwj commented May 20, 2019

Uh oh!

maflcko commented May 20, 2019

Uh oh!

dongcarl commented May 20, 2019

Uh oh!

Kemu commented May 20, 2019

Uh oh!

Kemu commented May 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented May 21, 2019

Conflicts

Uh oh!

jonasschnelli commented May 21, 2019

Uh oh!

practicalswift commented May 21, 2019

Uh oh!

dongcarl commented May 21, 2019

Uh oh!

DrahtBot commented May 22, 2019

Uh oh!

Uh oh!

jonatack commented May 20, 2019 •

edited

Loading

Kemu commented May 20, 2019 •

edited

Loading