build: Finish up out-of-tree changes #8133

theuni · 2016-06-01T23:46:42Z

This fixes the remaining out-of-tree issues:

The entire leveldb dir was copied out
Python caches were left hanging around
A qt file was left dangling in the build-dir
The rpc tests couldn't be run
OSX's make deploy was broken

make distcheck now passes, and Travis has been switched to an out-of-tree build for dogfooding purposes. Fingers-crossed that it works.

Don't glob the leveldb for dist. That means we need to enumerate the headers.

- clear the __pycache__ during 'make clean' - Copy the qrc locale file to a temp location and remove it when finished (rcc expects everything to be in the same path)

- Link pull-tester/rpc-tests.py to the build dir - Add the build-dir's config to the python path so that tests can find it - The tests themselves are in srcdir - Clean up __pycache__ in 'make clean'

The plist is generated, lives in builddir.

maflcko · 2016-06-02T17:11:21Z

qa/pull-tester/rpc-tests.py

@@ -37,7 +38,7 @@
    # terminal via ANSI escape sequences:
    BOLD = ('\033[0m', '\033[1m')

-RPC_TESTS_DIR = BUILDDIR + '/qa/rpc-tests/'
+RPC_TESTS_DIR = SRCDIR + '/qa/rpc-tests/'


Note to myself: Does the cache (from create_cache.py et al.) end up in SRCDIR now?

Wasn't it always (and still is) the case that the cache ends up in the current directory when the tests are launched?

Correct.

Though, this should be changed. Having different caches lying around in folders that happen to be your working dir right then is misleading at best.

Anyway, this is unrelated to this pull.

I'm having a hard time answering that question because of the symlink interaction with Python. builddir/rpc-tests.py is now a symlink to sourcedir/rpc-tests.py because it needs to be writable. But that makes the definition of "current dir" hazy.

@MarcoFalke I suppose it would be best to make the location explicit.

Sure, this is on my list of cleanups. If no one beats me to it, you can expect it some time after feature freeze. ;)

maflcko · 2016-06-02T17:11:35Z

Concept ACK

theuni · 2016-06-02T17:57:08Z

Generally speaking, if it's used as-is from git, it's srcdir. If it's generated or pre-processed, it's builddir.

Put another way: after checkout, think of srcdir as read-only. If anything is modified, it must happen in builddir.

laanwj · 2016-06-03T13:24:01Z

utACK 142ffc7

maflcko · 2016-06-03T15:39:13Z

make check fails with:

Running test/bitcoin-util-test.py...
Traceback (most recent call last):
  File "../../src/test/bitcoin-util-test.py", line 8, in <module>
    import buildenv
ImportError: bad magic number in 'buildenv': b'\x03\xf3\r\n'

theuni · 2016-06-03T15:44:58Z

@MarcoFalke Yes, I just saw that as well while trying to answer the question above. I'm not sure why Travis passes.

I'll fix that one up.

theuni · 2016-06-03T18:16:45Z

Pushed a quick fix for @MarcoFalke's problem. I'm not sure if we really need that or not, but it's probably better than each dev running into the problem and googling for themselves.

The issue is this: previous invocations of python2 in an in-tree build would leave a buildenv.pyc hanging around in srcdir. We prepend python's search path with builddir, but for some reason it ends up finding the srcdir/buildenv.pyc (and missing .py) before the builddir/buildenv.py. python3 refuses to load the old python2 .pyc, and errors because it can't find a .py at the same path for recompilation.

With python3, pyc's are organized into a pycache dir, and tagged like: pycache/buildenv.cpython-35.pyc. So it should not be possible to hit this case anymore.

…builds This was caused by an pyc files hanging around from previous python2 invocations, when the matching .py missing from that path. This should not be a problem with python3's tagged caches.

maflcko · 2016-06-03T18:48:29Z

Tested ACK 340012d

maflcko · 2016-06-03T18:53:16Z

qa/pull-tester/rpc-tests.py

@@ -29,6 +29,7 @@
 import tempfile
 import re

+sys.path.append("qa/pull-tester/")


Is it safe to assume everyone runs the tests from the root of buildir?

Probably not, but FWIW at least I do...

Heh, I run them from qa/pull-tester. I had no idea you could run them from the root...

Not sure how to fix this...

What about byte-compiling the rpc-tests.py instead of linking?

Heh, I run them from qa/pull-tester. I had no idea you could run them from the root...

Same here. But running from the root is ok with me.

luke-jr · 2016-06-05T17:38:43Z

Extensively-tested, with the following results:

I built, make check'd, and confirmed the reported version numbers for:

Before this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a clean git clone: v0.12.99.0-fd89d8e
Before this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a clean dist dir: v0.12.99.0-gfd89d8e
Before this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a clean dist dir under a foreign git checkout: v0.12.99.0-6862d12
With this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a clean git clone: v0.12.99.0-08cb06e
With this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a subdirectory of a clean git clone owned by another user: v0.12.99.0-08cb06e-dirty
With this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a parallel directory of a clean git clone owned by another user, under a foreign git checkout: v0.12.99.0-08cb06e-dirty
With this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a clean dist dir under a foreign git checkout: v0.12.99.0-6862d12
With this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a clean dist dir: v0.12.99.0-g08cb06e
With this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a subdirectory of a clean dist dir owned by another user: v0.12.99.0-g08cb06e
With this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522), built in a parallel directory of a clean dist dir owned by another user: v0.12.99.0-g08cb06e

Note that:

08cb06e = With this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522)
fd89d8e = Before this PR (plus Bugfix: Only use git for build info if the repository is actually the right one #7522)
6862d12 = Commit in empty foreign git repo

Observations:

No change to previously-supported build scenarios.
Dist dir builds within a foreign git repository incorrectly use the foreign git repo's commit hash in the version. This is already a problem (even with Bugfix: Only use git for build info if the repository is actually the right one #7522 apparently!), so not a blocker for this PR.
Dist dir builds prefix the commit hash with 'g', while git builds do not. This is already a problem, so also not a regression in this PR.
Clean git builds using a different build dir incorrectly tag the version as "-dirty". This should probably be fixed in this PR.

laanwj · 2016-06-06T08:17:26Z

BTW seems make translate also needs changes to work out-of-tree. Not necessarily in this pull.

laanwj · 2016-06-06T08:23:04Z

Looks like this is enough:

diff --git a/src/Makefile.qt.include b/src/Makefile.qt.include
index 3b39919..327a3d1 100644
--- a/src/Makefile.qt.include
+++ b/src/Makefile.qt.include
@@ -390,13 +390,13 @@ QT_QM=$(QT_TS:.ts=.qm)

 SECONDARY: $(QT_QM)

-qt/bitcoinstrings.cpp: $(libbitcoin_server_a_SOURCES) $(libbitcoin_wallet_a_SOURCES)
+$(srcdir)/qt/bitcoinstrings.cpp: $(libbitcoin_server_a_SOURCES) $(libbitcoin_wallet_a_SOURCES)
        @test -n $(XGETTEXT) || echo "xgettext is required for updating translations"
        $(AM_V_GEN) cd $(srcdir); XGETTEXT=$(XGETTEXT) PACKAGE_NAME="$(PACKAGE_NAME)" COPYRIGHT_HOLDERS="$(COPYRIGHT_HOLDERS)" COPYRIGHT_HOLDERS_SUBSTITUTION="$(COPYRIGHT_HOLDERS_SUBSTITUTION)" $(PYTHON) ../share/qt/extract_strings_qt.py $^

-translate: qt/bitcoinstrings.cpp $(QT_FORMS_UI) $(QT_FORMS_UI) $(BITCOIN_QT_CPP) $(BITCOIN_QT_H) $(BITCOIN_MM)
+translate: $(srcdir)/qt/bitcoinstrings.cpp $(QT_FORMS_UI) $(QT_FORMS_UI) $(BITCOIN_QT_CPP) $(BITCOIN_QT_H) $(BITCOIN_MM)
        @test -n $(LUPDATE) || echo "lupdate is required for updating translations"
-       $(AM_V_GEN) QT_SELECT=$(QT_SELECT) $(LUPDATE) $^ -locations relative -no-obsolete -ts qt/locale/bitcoin_en.ts
+       $(AM_V_GEN) QT_SELECT=$(QT_SELECT) $(LUPDATE) $^ -locations relative -no-obsolete -ts $(srcdir)/qt/locale/bitcoin_en.ts

 $(QT_QRC_LOCALE_CPP): $(QT_QRC_LOCALE) $(QT_QM)
        @test -f $(RCC)

Updated: bitcoin_en.ts was written to the build directory instead of the source directory, now fixed.

Also available at: https://github.com/laanwj/bitcoin/tree/2016_06_translate_out_of_tree

theuni · 2016-06-06T13:34:53Z

@luke-jr Whoa, that's great feedback! Thanks for the testing. I'm having a look at the "-dirty" issue.

@laanwj Thanks, I'll add that here.

laanwj · 2016-06-09T05:25:08Z

Let's try to get this merged soon.
I can PR the make translate change separately if that's mroe convenient.

theuni · 2016-06-09T20:50:15Z

@luke-jr I believe I've found the issue here. When you "chown -R root:root .", you're setting the git index to read-only. That makes "git diff" fail to update the index, which makes the diff-index fail.

For the sake of this PR, I'm ok with that not working correctly, since git is in a pretty bad state.

laanwj · 2016-06-10T08:01:10Z

since git is in a pretty bad state

That sounds ominous. Time to switch to mercurial or bzr? :-)

laanwj · 2016-06-10T08:04:50Z

"works for me" ACK d1a3d57

d1a3d57 bulid: fix "make translate" when out-of-tree (Cory Fields) 340012d build: add temporary fix for "bad magic number" error in out-of-tree builds (Cory Fields) 142ffc7 travis: use out-of-tree build (Cory Fields) 92e37a3 build: fix out-of-tree 'make deploy' for osx (Cory Fields) ab95d5d build: a few ugly hacks to get the rpc tests working out-of-tree (Cory Fields) fc4ad0c build: more out-of-tree fixups (Cory Fields) 0cb0f26 build: out-of-tree fixups (Cory Fields)

theuni · 2016-06-10T08:41:13Z

Haha. That was a half-complete thought as usual.

Should read: "I'm ok with that not working correctly, since if you're hitting this case, your git repo is already unreliable".

d1a3d57 bulid: fix "make translate" when out-of-tree (Cory Fields) 340012d build: add temporary fix for "bad magic number" error in out-of-tree builds (Cory Fields) 142ffc7 travis: use out-of-tree build (Cory Fields) 92e37a3 build: fix out-of-tree 'make deploy' for osx (Cory Fields) ab95d5d build: a few ugly hacks to get the rpc tests working out-of-tree (Cory Fields) fc4ad0c build: more out-of-tree fixups (Cory Fields) 0cb0f26 build: out-of-tree fixups (Cory Fields)

Update LevelDB to upstream commit f545dfabf Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#7911 - bitcoin/bitcoin#7982 - bitcoin/bitcoin#8133 - bitcoin/bitcoin#8784 - Only the missing changes. - bitcoin/bitcoin#8826 - bitcoin/bitcoin#8613 - bitcoin/bitcoin#10544 - bitcoin/bitcoin#10633 - Only the changes to files and code we have. - bitcoin/bitcoin#10806 - bitcoin/bitcoin#10958 - bitcoin/bitcoin#12451 - bitcoin/bitcoin#13925 - bitcoin/bitcoin#15270 This upgrades LevelDB in the exact same commit progression as upstream, up to January 2019.

theuni force-pushed the out-of-tree-clean branch from 5ba8891 to 80f68b7 Compare June 2, 2016 00:25

theuni added 3 commits June 1, 2016 20:31

build: out-of-tree fixups

0cb0f26

Don't glob the leveldb for dist. That means we need to enumerate the headers.

build: more out-of-tree fixups

fc4ad0c

- clear the __pycache__ during 'make clean' - Copy the qrc locale file to a temp location and remove it when finished (rcc expects everything to be in the same path)

build: a few ugly hacks to get the rpc tests working out-of-tree

ab95d5d

- Link pull-tester/rpc-tests.py to the build dir - Add the build-dir's config to the python path so that tests can find it - The tests themselves are in srcdir - Clean up __pycache__ in 'make clean'

theuni force-pushed the out-of-tree-clean branch 2 times, most recently from c2c7a4d to 142ffc7 Compare June 2, 2016 02:14

theuni added 2 commits June 1, 2016 22:20

build: fix out-of-tree 'make deploy' for osx

92e37a3

The plist is generated, lives in builddir.

travis: use out-of-tree build

142ffc7

jonasschnelli added the Build system label Jun 2, 2016

maflcko reviewed Jun 2, 2016
View reviewed changes

theuni force-pushed the out-of-tree-clean branch from e5ccf56 to 340012d Compare June 3, 2016 18:42

build: add temporary fix for "bad magic number" error in out-of-tree …

340012d

…builds This was caused by an pyc files hanging around from previous python2 invocations, when the matching .py missing from that path. This should not be a problem with python3's tagged caches.

maflcko reviewed Jun 3, 2016
View reviewed changes

laanwj mentioned this pull request Jun 6, 2016

Backport leveldb build integration to 0.12 #8148

Merged

laanwj added this to the 0.13.0 milestone Jun 9, 2016

bulid: fix "make translate" when out-of-tree

d1a3d57

laanwj merged commit d1a3d57 into bitcoin:master Jun 10, 2016

theuni deleted the out-of-tree-clean branch June 10, 2016 08:43

laanwj mentioned this pull request Jun 14, 2016

Remaining minor issues for out-of-tree build #7937

Closed

3 tasks

luke-jr mentioned this pull request Jun 17, 2016

Bugfix: Only use git for build info if the repository is actually the right one #7522

Merged

jmcorgan mentioned this pull request Jun 29, 2016

Backport remaining commits for out-of-tree builds from master to 0.12 branch #8284

Closed

maflcko mentioned this pull request Jan 11, 2017

build: fix qt distdir builds (retry) #9513

Merged

jnewbery mentioned this pull request Jan 31, 2017

Improve rpc-tests.py #9657

Merged

str4d mentioned this pull request Sep 22, 2020

Update LevelDB to upstream commit f545dfabf zcash/zcash#4739

Merged

str4d mentioned this pull request Nov 21, 2020

Backport migration from rpc-tests.sh to rpc-tests.py zcash/zcash#4874

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

build: Finish up out-of-tree changes #8133

build: Finish up out-of-tree changes #8133

Uh oh!

Conversation

theuni commented Jun 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maflcko commented Jun 2, 2016

Uh oh!

theuni commented Jun 2, 2016

Uh oh!

laanwj commented Jun 3, 2016

Uh oh!

maflcko commented Jun 3, 2016

Uh oh!

theuni commented Jun 3, 2016

Uh oh!

theuni commented Jun 3, 2016

Uh oh!

maflcko commented Jun 3, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maflcko Jun 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luke-jr commented Jun 5, 2016

Uh oh!

laanwj commented Jun 6, 2016

Uh oh!

laanwj commented Jun 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theuni commented Jun 6, 2016

Uh oh!

laanwj commented Jun 9, 2016

Uh oh!

theuni commented Jun 9, 2016

Uh oh!

laanwj commented Jun 10, 2016

Uh oh!

laanwj commented Jun 10, 2016

Uh oh!

theuni commented Jun 10, 2016

Uh oh!

Uh oh!

theuni commented Jun 1, 2016 •

edited

Loading

maflcko Jun 4, 2016 •

edited

Loading

laanwj commented Jun 6, 2016 •

edited

Loading