Autotools buildsystem #2943
Autotools buildsystem #2943
Conversation
|
ACK |
|
I believe I've whipped the gitian weirdness into shape enough for initial merge. Only real blockers as I see now are acks from various platform devs (osx, win32 cross, win32 native), and docs to accompany. I'm afraid I've reached my limit now, as far as those things are concerned. Awaiting some input, though there's no hurry from my side. |
|
Doesn't work for me on OSX 10.7.5 with MacPorts autotools: If I ignore the 'failed with exit status 1' and run ./configure, I get 'configure: error: "Big Endian not supported"' |
|
I grabbed automake 1.14 and tested locally. I just veified/pushed the fix for that problem. New parts for my Macbook should be here tomorrow, Monday at the latest. I can test definitively then if you don't feel like being a guinea pig. |
|
Looks like the problem is trying to build for osx 10.5 from 10.7, which isn't going to happen without a good amount of local fiddling (likely installing xcode 3). You might be able to get past it for now by disabling the release compatibility flags: ./configure --enable-debug |
|
@theuni Do you use MacPorts, HomeBrew or something else on OSX? |
|
Just pushed a boatload of fixes. Sorry for yet another round of changes, but this has turned out to be much more complicated than originally anticipated. I can squash down before merge, but I decided to push individual changes in case anyone is following the progress. Fully verified on osx now that my macbook is up and running again (yay!). I also scrapped the 'windows-deploy' target and replaced it with the generic 'deploy' one. 'make deploy' on osx will produce a fully-functional dmg. When building for windows, 'make deploy' will create a fully-functional win32 installer. At this point, it's feeling very very solid. That said, I only have osx 10.6 at my disposal currently, so it hasn't been tested on 10.7/10.8 yet. @fanquake: Tested extensively with macports and homebrew after today's changes. The build procedure is pretty much: install macports/homebrew depends as before, but with automake/autoconf added. Additionally, you can install qt directly from macports/homebrew and it should just work (tm). After that, standard configure+make should be all it takes. I'll try to get the osx docs written+pushed tomorrow. |
|
ACK code changes, and seems to work fine on Ubuntu (13.04). @laanwj still wanted some improvement to the .pro file, though? |
|
So... ... with MacPorts openssl in /opt/local/{include,lib} |
|
@gavinandresen Usually I do something like CXXFLAGS="-O2 -Wall -g -I/opt/local/include" LDFLAGS="-L/opt/local/lib" ./configure |
|
Still working on the osx docs.. I spent all day working out possible configs from 10.6-10.8 (managed to get my hands on a 10.8 macbook), both macports and brew. Short answer: sounds like you need to 'sudo port install pkgconfig'. After that, it should find everything for you. |
|
10.5 is officially dropped? (Apple has.) |
|
No, I don't have a 10.5 machine at my disposal. However, release binaries are still built as 32-bit 10.5 compatible after my changes. Dropping 10.5 would make lots of things much easier, but I don't wish to mix a policy change (which OSs are supported) with a technical change (buildsystem switch) for fear that this would needlessly delay the process. |
|
port install pkgconfig worked nicely. I'd volunteer to help write documentation, but, as you can tell, I don't know enough about autotools to be helpful. |
|
No worries. The reason I'm hesitant to do the docs is because it essentially needs a rm -f *. The point of autotools is abstraction, so there's really no need for platform-specific instructions. There's also really no need for distinction between qt/non-qt builds anymore. My intention is essentially:
The problem with the above is that I need a good bit of feedback. I've been looking at this (and working on other autotools projects) for so long that it's become hard to tell what's non-obvious. I don't think I'll receive much feedback until it's merged and there are more devs exposing its warts. So my proposal is to do a quick readme with monkey-style copy/paste commands for the various platforms, and let that suffice until merge. Post-merge, the above outline could be much more easily worked out. Thoughts? |
|
I would propose low merge hurdles, namely
Anything else can and should be resolved in-tree. Once the system is working "en grosse" merge it, leaving sector experts and motivated parties to hammer out the final details. In particular, Qt will likely need additional attention after merge, it sounds like. That's OK. We are not going to have 0.9-rc1 tomorrow or next week. The first three seem to be well on their way to being completed. The last is the toughie, IMO: need to get our pulltester back in working order almost immediately after merge, so we need to beg BlueMatt for some time or hopefully grow some more pulltester experts (I am entirely hands-off Java, not being a fan of the language). tl;dr: Let's not endlessly iterate this outside of 'master'. Merge it. |
|
Whew, that's great to hear. Getting it merged will will really help. I've been working on pulltester actually, and I"m nearly there. I've add lcov support to the buildsystem, as well as support for the java test that pulltester runs (side-rant: I find it troubling when these things are out-of-tree and firmly believe that they should be testable by every dev). So Pulltester essentially becomes a few configure/make and that's it. I believe I can have that pushed up tonight. |
|
Ok, that should do it for the pull tester. I've integrated lcov and the java test into the build. Also added a set of scripts for contrib that pull-tester can run from jenkins. The big benefit is that everything is now in-tree, so if they change in the future, there's no worry of getting out of sync with the build procedure. Additionally, devs can run the script and yield the same result as the bot. @TheBlueMatt ping. With any luck, the bot just needs to run test-scripts/pull-tester.sh . That in-turn kicks off https://github.com/theuni/bitcoin/blob/dd08a1146f5408af5316651fbd89027575bedf14/contrib/test-scripts/build-tests.sh.in, which is as close to the old behavior as I could get it. |
|
Nice! If Matt doesn't get to it before me, I'll teach the pull-tester machine to Do The Right Thing (if test-scripts/pull-tester.sh exists, run it. Else do what we do now). |
|
Sounds good. It will need a little more love, as it will fail until the win32 deps are updated to the new version. The old ones were scattered around, the new ones are in a prefixed layout. IMO, that should wait until after this is merged, and just let it fail until then. The old/new win32 deps can coexist to facilitate build-testing of the old kind. There will just need to be 2 copies, one in /mnt/mingw as they currently are, and another set somewhere else. If running the new script, it should be executed as: |
|
@gavinandresen If/when you go to update the bot, the params have changed to accommodate the out-of-tree java tool. It should now be invoked as: |
|
Ok, I've pushed the last of my changes. I now consider this to be complete. I'm sure it's full of bugs and facepalms, but I think it now meets @jgarzik's criteria above. pull-tester should be working pending teaching the bot its new tricks, and a rebase of this branch. The win32 build will fail until it gets updated deps, so I've disabled that build in the test-scripts for now. Any objections to squashing this down now and rebasing to master? |
|
Squashed down and ready from my side, with the understanding that it will probably require some patience for a few days/weeks. The individual commits (plus a few fixes related to rebasing to master) can be seen here if needed: https://github.com/theuni/bitcoin/commits/autotools-final-pre-rebase I need to step away from this for the rest of the week, but I'm happy to answer any questions that arise. |
|
Nice work, I tested around a bit and it worked wonderfully. I first thought it was caused by parallel build, but it happens with single-threaded build as well. |
|
Yea, out-of-tree builds can't really work without some serious hackery, which I've declined to add so far. You can see how it would work with 'make distcheck', which runs an out-of-tree build. The "solution" applied there is to copy leveldb to the build dir. |
|
squashed with the pull-tester updates and 2 additional fixes:
|
|
Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/35b8af92265ed74de63c3818e5290c27b3f35df2 for binaries and test log. |
|
Won't work for me on OSX 10.7.2, get an error in configure. |
|
Also not working for me, as it isn't looking for bdb headers in the standard paths :( |
|
"standard path" varies even between Linux OS's, because BDB 4.x by default just installs into /usr/include/db.h. |
|
@shripadk I think I see it. (Also, you need to install pkg-config from ports). Could some committer kill the extra ] that snuck in here: https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L411 |
|
/usr/include/features.h:330:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp] This warning prints very often with a plain ./configure. The default builds without any -O. Is this intended?
|
|
Not intended. Now that this has been merged, it's worth a bit of debate as to what reasonable defaults should be for debug/release configurations should be. Before, they were different per OS. I'd say that for debug, "-g -O2" probably makes sense. |
|
I find -O2 makes debugging impractical, so I personally prefer "-ggdb -O0", but if -O1 is needed for security that's probably good enough (besides, -O0 doesn't even inline :p). |
|
Getting error while trying to build with Qt. I tried |
follow up of 35b8af9 (bitcoin#2943)
follow up of 35b8af9 (bitcoin#2943)
follow up of 35b8af9 (bitcoin#2943)
Refactor validation signals for ChainLocks, InstantSend and PrivateSend to get rid of SyncTransaction
f52fafc build: Drop pointless sed commands (Hennadii Stepanov) Pull request description: Since moving to Autotools build system (35b8af9, #2943, 2013-09), tag strings created by Qt specialized compilers ([uic](https://doc.qt.io/qt-5/uic.html), [moc](https://doc.qt.io/qt-5/moc.html), [rcc](https://doc.qt.io/qt-5/rcc.html)) were being removed. A bit later (70c71c5, #4241, 2014-06) this rule was dropped for the uic, and since then all of the generated `ui_*.h` files contain the following string: ``` ** Created by: Qt User Interface Compiler version 5.12.8 ``` Such strings do not contain any timestamps, and cannot cause any non-determinism. The removing of them seems pointless. Diffs for some files: ```diff --- master/intro.moc +++ pr/intro.moc @@ -1,6 +1,7 @@ /**************************************************************************** ** Meta object code from reading C++ file 'intro.cpp' ** +** Created by: The Qt Meta Object Compiler version 67 (Qt 5.12.8) ** ** WARNING! All changes made in this file will be lost! *****************************************************************************/ ``` ```diff --- master/moc_addressbookpage.cpp +++ pr/moc_addressbookpage.cpp @@ -1,6 +1,7 @@ /**************************************************************************** ** Meta object code from reading C++ file 'addressbookpage.h' ** +** Created by: The Qt Meta Object Compiler version 67 (Qt 5.12.8) ** ** WARNING! All changes made in this file will be lost! *****************************************************************************/ ``` ```diff --- master/qrc_bitcoin.cpp +++ pr/qrc_bitcoin.cpp @@ -1,6 +1,7 @@ /**************************************************************************** ** Resource object code ** +** Created by: The Resource Compiler for Qt version 5.12.8 ** ** WARNING! All changes made in this file will be lost! *****************************************************************************/ ``` ACKs for top commit: laanwj: ACK f52fafc Tree-SHA512: 31f5c19b37645b4914f17d8c234b7ae8781a0499c4b250ffef07d70b7552954fb682f58a75d76162f98ab5e1667288b3a041df2705573fb00523e87b9c1fd47f
Continuation of #2805.
Changes since Round 2:
Rebased on current master
Big cleanups
Added protoc/libprotobuf support
Added win32 installer make target
Added Gitian descriptors
I could use some help with the following items:
I've briefly updated the documentation for unix. Sadly, my macbook busted last weekend (should be fixed in ~1wk), So I was not able to add the osx dmg build target as I had intended, nor the documentation for building on osx.
Is anyone willing to help me out with the above?
Here is the final go at an Autotools build system replacement. It is meant to be a drop-in replacement for the current system(s), providing the same features with no net changes. It can also live side-by-side with the old system while sharing the same build-related variables in order to facilitate a smooth transition.
I hope the benefits are obvious enough: A single/shared build procedure, portability, ease of packaging, ease for downstreams, ease for repository maintainers, cross-compilation, etc. I don't vouch for Autotools in any way, in fact, this configure.am is downright ugly (mainly just because of mingw though), but it's portable and well-established.
This does away with the need for qmake, as the Makefile is capable of generating everything it needs in a portable way.
Building from CLI:
For Linux, assuming the dependencies have been met, the build procedure looks like this:
Same for OS X, but the pkg-config path needs to be hooked up from macports first:
For mingw it's the same, but you will need to provide lots of paths in the form of:
In addition, there are helpers for qt and boost to help with finding some locations.
Use ./configure --help to see the available options.
Native windows built is untested, as I don't have a windows environment at my disposal.
'make check' will run the unit tests and print the results.
I've done my best to avoid adding any new behavior or features, and I would much prefer to aim for feature-parity before making any improvements.