-
Notifications
You must be signed in to change notification settings - Fork 37.7k
build: Propagate well-known vars into depends #18820
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build: Propagate well-known vars into depends #18820
Conversation
Ref #15914. |
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsNo conflicts as of last run. |
Concept ACK Having this merged would allow enabling MemorySanitizer (MSan) more easily (#18288). |
@dongcarl It seems like the Travis macOS 10.12 job failed :) |
Concept ACK |
As of Boost 1.72.0, the darwin toolset has been removed from Boost Build, in favour of the Clang toolset; see this commit: boostorg/build@eb8da52. The reasoning (not given in the commit) being that darwin was basically a copy of the clang toolset anyways. Some comments from the commit author here: boostorg/build#528. This change may also resolve the macOS issues seen in bitcoin#18820.
Tested a0e7fe7 on Linux Mint 19.3:
|
Concept ACK (assuming it solves my problems) |
Concept ACK. |
Seems like the issue is that we need to separate the |
a0e7fe7
to
645e080
Compare
Done, should work now. Glad to know that our build system is not the only one with weird idiosyncrasies 😬 |
ACK 645e080 -- patch looks correct and Travis is happy :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested 645e080 on Linux Mint 19.3 (x86_64):
$ make -C depends CC=clang CXX=clang++
$ CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure
...
CC = /usr/bin/ccache clang
...
CXX = /usr/bin/ccache clang++ -std=c++11
...
$ make clean && make
...
/usr/bin/ld: /home/hebasto/GitHub/bitcoin/depends/x86_64-pc-linux-gnu/share/../lib/libdb_cxx-4.8.a(cxx_db.o): relocation R_X86_64_32S against symbol `_ZTV2Db' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: /home/hebasto/GitHub/bitcoin/depends/x86_64-pc-linux-gnu/share/../lib/libdb_cxx-4.8.a(cxx_env.o): relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: /home/hebasto/GitHub/bitcoin/depends/x86_64-pc-linux-gnu/share/../lib/libdb_cxx-4.8.a(cxx_except.o): relocation R_X86_64_32S against symbol `_ZTV11DbException' can not be used when making a PIE object; recompile with -fPIC
# loads of "recompile with -fPIC" link errors
Maybe instead of backporting patch just bump FreeType version to the latest 2.10.2 where that patch is included? |
Can you check your logs to see the configuration logs of |
output
|
@dongcarl
|
REMOVED |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review ACK f0d7ed1. Changes since last review: adding comment explaining check for predefined make variables, dropping freetype commit, adding commit whitelisting overrides for recursive makes
Please don't merge this just yet. There's at least one build failure that will need addressing (likely in a follow up, but I want to understand it before merging). |
There is a at least one failure building make -C depends/ -j6 V=1 HOST=x86_64-apple-darwin16 CC=clang CXX=clang++
....
echo Configuring zlib...
Configuring zlib...
rm -rf /bitcoin/depends/x86_64-apple-darwin16; mkdir -p /bitcoin/depends/x86_64-apple-darwin16/lib; cd /bitcoin/depends/x86_64-apple-darwin16; tar --no-same-owner -xf /bitcoin/depends/built/x86_64-apple-darwin16/native_cctools/native_cctools-3764b223c011574971ee3ae09ce968ba5dc2f00f-d41d83b91be.tar.gz;
mkdir -p /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/.
cd /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/.; PKG_CONFIG_LIBDIR=/bitcoin/depends/x86_64-apple-darwin16/lib/pkgconfig PKG_CONFIG_PATH=/bitcoin/depends/x86_64-apple-darwin16/share/pkgconfig CMAKE_MODULE_PATH=/bitcoin/depends/x86_64-apple-darwin16/lib/cmake PATH=/bitcoin/depends/x86_64-apple-darwin16/native/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin env CC="clang" CFLAGS="-pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC" RANLIB="x86_64-apple-darwin16-ranlib" AR="x86_64-apple-darwin16-ar" AR="x86_64-apple-darwin16-libtool" ARFLAGS="-o" ./configure --static --prefix=/bitcoin/depends/x86_64-apple-darwin16
Building static library libz.a version 1.2.11 with clang.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
touch /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/./.stamp_configured
echo Building zlib...
Building zlib...
mkdir -p /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/.
cd /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/.; PATH=/bitcoin/depends/x86_64-apple-darwin16/native/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin make libz.a
make[1]: Entering directory '/bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c'
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o adler32.o adler32.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o crc32.o crc32.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o deflate.o deflate.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o infback.o infback.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inffast.o inffast.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inflate.o inflate.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o inftrees.o inftrees.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o trees.o trees.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o zutil.o zutil.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o compress.o compress.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o uncompr.o uncompr.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzclose.o gzclose.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzlib.o gzlib.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzread.o gzread.c
clang -pipe -O2 -I/bitcoin/depends/x86_64-apple-darwin16/include -fPIC -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -c -o gzwrite.o gzwrite.c
x86_64-apple-darwin16-libtool -o libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o
error: x86_64-apple-darwin16-libtool: file: adler32.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: crc32.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: deflate.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: infback.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: inffast.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: inflate.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: inftrees.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: trees.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: zutil.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: compress.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: uncompr.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: gzclose.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: gzlib.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: gzread.o is not an object file (not allowed in a library)
error: x86_64-apple-darwin16-libtool: file: gzwrite.o is not an object file (not allowed in a library)
make[1]: *** [Makefile:131: libz.a] Error 1
make[1]: Leaving directory '/bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c'
make: *** [funcs.mk:262: /bitcoin/depends/work/build/x86_64-apple-darwin16/zlib/1.2.11-9d169cb6e6c/./.stamp_built] Error 2
make: Leaving directory '/bitcoin/depends' I assume this is because we lose our macOS flags. i.e @dongcarl thoughts? If you want to address this in a follow up I think that's reasonable. |
I'll also just mention that you can't seem to cross-compile for Windows when using Clang. The build bombs out at Boost (haven't looked into this at all 😴): echo Building boost...
Building boost...
mkdir -p /bitcoin/depends/work/build/x86_64-w64-mingw32/boost/1_70_0-d650037a261/.
cd /bitcoin/depends/work/build/x86_64-w64-mingw32/boost/1_70_0-d650037a261/.; PATH=/bitcoin/depends/x86_64-w64-mingw32/native/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ./b2 -d2 -j2 -d1 --prefix=/bitcoin/depends/work/staging/x86_64-w64-mingw32/boost/1_70_0-d650037a261/bitcoin/depends/x86_64-w64-mingw32 --layout=tagged --build-type=complete --user-config=user-config.jam threading=multi link=static -sNO_BZIP2=1 -sNO_ZLIB=1 variant=release target-os=windows binary-format=pe threadapi=win32 runtime-link=static address-model=64 toolset=clang stage
Performing configuration checks
- default address-model : 64-bit (cached)
- default architecture : x86 (cached)
- BOOST_COMP_GNUC >= 4.3.0 : no (cached)
Component configuration:
- atomic : not building
- chrono : not building
- container : not building
- context : not building
- contract : not building
- coroutine : not building
- date_time : not building
- exception : not building
- fiber : not building
- filesystem : building
- graph : not building
- graph_parallel : not building
- headers : not building
- iostreams : not building
- locale : not building
- log : not building
- math : not building
- mpi : not building
- program_options : not building
- python : not building
- random : not building
- regex : not building
- serialization : not building
- stacktrace : not building
- system : building
- test : building
- thread : building
- timer : not building
- type_erasure : not building
- wave : not building
...patience...
...patience...
...found 1632 targets...
...updating 5 targets...
clang-linux.compile.c++.without-pch bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread_primitives.obj
"clang++" -c -x c++ -fvisibility-inlines-hidden -std=c++11 -fvisibility=hidden -I/bitcoin/depends/x86_64-w64-mingw32/include -m64 -O3 -Wall -fvisibility=hidden -Wno-inline -pthread -Wextra -Wno-long-long -Wno-unused-parameter -Wno-variadic-macros -Wunused-function -DBOOST_ALL_NO_LIB=1 -DBOOST_CHRONO_STATIC_LINK=1 -DBOOST_THREAD_BUILD_LIB=1 -DBOOST_THREAD_USES_CHRONO -DBOOST_THREAD_WIN32 -DBOOST_USE_WINDOWS_H -DNDEBUG -DWIN32_LEAN_AND_MEAN -I"." -o "bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread_primitives.obj" "libs/thread/src/win32/thread_primitives.cpp"
In file included from libs/thread/src/win32/thread_primitives.cpp:10:
In file included from ./boost/winapi/dll.hpp:14:
./boost/winapi/basic_types.hpp:19:11: fatal error: 'windows.h' file not found
# include <windows.h>
^~~~~~~~~~~
1 error generated.
...failed clang-linux.compile.c++.without-pch bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread_primitives.obj...
clang-linux.compile.c++.without-pch bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread.obj
"clang++" -c -x c++ -fvisibility-inlines-hidden -std=c++11 -fvisibility=hidden -I/bitcoin/depends/x86_64-w64-mingw32/include -m64 -O3 -Wall -fvisibility=hidden -Wno-inline -pthread -Wextra -Wno-long-long -Wno-unused-parameter -Wno-variadic-macros -Wunused-function -DBOOST_ALL_NO_LIB=1 -DBOOST_CHRONO_STATIC_LINK=1 -DBOOST_THREAD_BUILD_LIB=1 -DBOOST_THREAD_USES_CHRONO -DBOOST_THREAD_WIN32 -DBOOST_USE_WINDOWS_H -DNDEBUG -DWIN32_LEAN_AND_MEAN -I"." -o "bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread.obj" "libs/thread/src/win32/thread.cpp"
In file included from libs/thread/src/win32/thread.cpp:11:
In file included from ./boost/thread/thread_only.hpp:15:
In file included from ./boost/thread/win32/thread_data.hpp:11:
In file included from ./boost/thread/win32/thread_primitives.hpp:18:
./boost/detail/interlocked.hpp:53:11: fatal error: 'windows.h' file not found
# include <windows.h>
^~~~~~~~~~~
1 error generated.
...failed clang-linux.compile.c++.without-pch bin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden/win32/thread.obj...
...skipped <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>libboost_thread-mt-s-x64.lib(clean) for lack of <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>win32/thread.obj...
...skipped <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>libboost_thread-mt-s-x64.lib for lack of <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>win32/thread.obj...
...skipped <p/bitcoin/depends/work/build/x86_64-w64-mingw32/boost/1_70_0-d650037a261/stage/lib>libboost_thread-mt-s-x64.lib for lack of <pbin.v2/libs/thread/build/clang-linux-7.0.1/release/link-static/target-os-windows/threadapi-win32/threading-multi/visibility-hidden>libboost_thread-mt-s-x64.lib...
...failed updating 2 targets...
...skipped 3 targets...
make: *** [funcs.mk:262: /bitcoin/depends/work/build/x86_64-w64-mingw32/boost/1_70_0-d650037a261/./.stamp_built] Error 1
make: Leaving directory '/bitcoin/depends' |
There is also a build failure on a fresh Ubuntu focal vm:
Steps to reproduce:
|
I think this PR by itself is already an improvement over the current situation (all cases that worked before still works, and a few cases that didn't work before now works) and I don't really want to overload this PR any longer. After doing some investigation, I think the following are worthwhile followups:
One thing to keep in mind for both of these followups is that we've been abusing @fanquake @MarcoFalke Let me know if that sounds okay. |
Is there documentation on how to pass CXXFLAGS into depends qt? I recently tried that and it appeared they were not picked up. |
Concept ACK on merging this |
We do some extremely nasty A cleanup of how we interact with QT's build system probably also deserves to be a followup |
@dongcarl followups sound ok. I think we've figured out a path to improving the macOS CFLAGS post this change as well. |
Gitian builds
|
f0d7ed1 depends: Propagate only specific CLI variables to sub-makes (Carl Dong) 0a33803 depends: boost: Use clang toolset if clang in CXX (Carl Dong) 1ce74bc depends: boost: Split target-os from toolset (Carl Dong) 2d4e480 depends: boost: Specify toolset to bootstrap.sh (Carl Dong) 3d6603e depends: Propagate well-known vars into depends (Carl Dong) Pull request description: From: bitcoin#18308 (comment) The following monstrosity is quite useful when invoked inside `depends`, and reviewers can use it to compare the behaviour of this change against master. ```bash make print-{{,{host,{,{i686,x86_64,riscv64}_}linux}_}{CC,CXX},boost_{cc,cxx}} ``` It would also be helpful to make sure that setting `HOST`, `CC`, and `CXX` does the right thing. The 3 hosts I found offered good coverage were: `{x86_64,i686,riscv64}-linux-gnu`. As we special-case the `x86_64` and `i686` hosts in `depends/hosts/linux.mk`, and `riscv64` is a sanity check for a non-special-cased host. ACKs for top commit: hebasto: ACK f0d7ed1, tested on Linux Mint 19.3 (x86_64): practicalswift: ACK f0d7ed1 -- patch looks correct laanwj: Code review and concept ACK f0d7ed1 ryanofsky: Code review ACK f0d7ed1. Changes since last review: adding comment explaining check for predefined make variables, dropping freetype commit, adding commit whitelisting overrides for recursive makes Tree-SHA512: b6b8e76f713c26a0add6cd685824e2f5639109236ee9f89338f7c79cb1b1f2c3897bfb62b80b023d6d1943b5a6eb282a2f827f1f499c5e556eca015d6635fa65
Summary: This will makes the boost depends build follow the CC/CXX environment variables (and allow the build with clang). Backport of core [[bitcoin/bitcoin#18820 | PR18820]]. Test Plan: Run the Gitian builds. Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D7672
Boost build backports Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#17087 - bitcoin/bitcoin#17231 - bitcoin/bitcoin#17928 - bitcoin/bitcoin#18820 - bitcoin/bitcoin#19764 Kudos to @dongcarl for all the excellent upstream depends system hackery!
de7766c Build: Update Boost download URL (Fuzzbawls) 7be66c9 Doc: document updated boost version in dependencies.md (Fuzzbawls) bcb77b6 depends: boost: Specify cflags+compileflags (Carl Dong) b8f8574 depends: boost: Remove unnecessary _archiver_ (Carl Dong) 29fdbd9 depends: boost: Cleanup toolset selection (Carl Dong) 28393b6 depends: boost: Cleanup architecture/address-model (Carl Dong) 6af3ffa depends: boost: Disable all compression (Carl Dong) 0f09788 depends: boost: Split into non-/native packages (Carl Dong) de97b06 depends: boost: Bump to 1.71.0 (Carl Dong) 19f474b depends: boost: Refer to version in URL (Carl Dong) 7d4257c depends: Propagate only specific CLI variables to sub-makes (Carl Dong) fcbf870 depends: boost: Use clang toolset if clang in CXX (Carl Dong) aad5009 depends: boost: Split target-os from toolset (Carl Dong) fae749b depends: boost: Specify toolset to bootstrap.sh (Carl Dong) c2bfedb depends: Propagate well-known vars into depends (Carl Dong) 091ae4a depends: Consistent use of package variable (Peter Bushnell) 635bdc1 depends: fix boost mac cross build with clang 9+ (Cory Fields) d796365 build: Add variable printing target to Makefiles (Carl Dong) Pull request description: Backports the following upstream PRs to clean up and update the Boost dependency - bitcoin#17087 - bitcoin#17231 - bitcoin#17928 - bitcoin#18820 - bitcoin#19764 ACKs for top commit: random-zebra: ACK de7766c furszy: no code changes after rebase, utACK de7766c and merging.. Tree-SHA512: 4abe88718892bce40a2df023e99a26a16ce3c5d470f55e70d6c6cca117ee8b8bb29968be6d40873bc9ece3f9df769bea248cbb38c1c5c2f318016702533f2736
merge bitcoin#18820, bitcoin#19764, bitcoin#13686, bitcoin#17538, bitcoin#18405: zmq backports, boost depends split
From: #18308 (comment)
The following monstrosity is quite useful when invoked inside
depends
, and reviewers can use it to compare the behaviour of this change against master.It would also be helpful to make sure that setting
HOST
,CC
, andCXX
does the right thing. The 3 hosts I found offered good coverage were:{x86_64,i686,riscv64}-linux-gnu
. As we special-case thex86_64
andi686
hosts independs/hosts/linux.mk
, andriscv64
is a sanity check for a non-special-cased host.