Cleanup code by using forward declarations and other methods... #2767

brandondahler · 2013-06-14T04:55:51Z

... of avoiding unnecessary header includes. Additionally alphabetizes and groups includes and forward declarations.

Note: there are a large amount of changes that have been rebased onto the most recent master. I did my best to ensure that no code was functionally changed; however, a code review of all my changes would be advised.

Theory

By using forward declarations, compilation speed can be increased in both incremental and full builds.

Forward declarations suffice over having the full declaration in cases where the type that is being forward declared is only used as a reference or being pointed to, and the usage is not an lvalue.

By making these modifications, the total number of header files needed to compile any .cpp file is reduced. Fewer header files being included reduces the total amount of code that needs to be compiled for each .cpp file.

Quantitative Results

How much time does this actually account for?

For bitcoind:

This spreadsheet shows detailed results for each of the different builds (note that there are multiple sheets in the spreadsheet), along with the interesting stats and graph for each.

This data was generated automatically via the scripts here.

Method

Go through each file, group each #include by type (see Order) and sort alphabetically
Using doxygen with the graphviz dots extension
- Go through each header, comment each #include individually and all headers that indirectly include the commented one
- See if it can build with the #include commented, analyze errors to understand if they can actually be solved with forward declares
- Remove, keep, or add a direct header to the required type, as needed.

Order

Most Generic

// License text as first
// lines

#if defined(HAVE_CONFIG_H)
#include "bitcoin-config.h"
#endif

#include "matching_file.h"

#include "local_file1.h"
#include "local_file2.h"

#include "file_in_other_dir1.h"
#include "file_in_other_dir2.h"

#include <system1.h>
#include <system2.h>

#ifdef SOMETHING1
#include <system3.h>
#endif

#include "json/a_file.h"
#include <library1.h>
#include <library2.h>

#ifdef SOMETHING2
#include <library3.h>
#endif

class ForwardDeclare1;
class ForwardDeclare2;

namespace SomeNamespace {
    class ForwardDeclare3;
}

...

Example1 - No matching file, no file_in_other_dir, no ifdefs

// License text as first
// lines

#include "local_file1.h"
#include "local_file2.h"

#include <system1.h>
#include <system2.h>

#include "json/a_file.h"
#include <library1.h>
#include <library2.h>

class ForwardDeclare1;
class ForwardDeclare2;

...

gmaxwell · 2013-06-14T05:03:05Z

And how do you prevent very bad thing from happening when one of the many duplicated definitions gets desynchronized?

brandondahler · 2013-06-14T05:12:51Z

Rebasing conflicts on the actual code changes made (one actual, moving some definitions from a header to the code file). Due diligence was taken on my part to ensure that the conflicts were resolved correctly, I would still recommend someone bite the bullet and look at all the changes that are not adding/removing/reordering header files.

Rebasing additionally conflicted on include files being added/removed usually because most of the includes were re-ordered, these were easy to resolve.

laanwj · 2013-06-14T08:46:41Z

@gmaxwell From what I understand this does not duplicate anything. It does pre-declare classes, but that only consists of the name and nothing more.

At first glance this seems like a lot of changes, but it's almost entirely restricted to the #include portion of files. The only substantial code move is moving WalletDB.h functions to its implementation file.

There's problems with compiling the tests after this, though (see pulltester output).

test/script_tests.cpp: In function 'CScript ParseScript(std::string)':
test/script_tests.cpp:47: error: 'replace_first' was not declared in this scope

sipa · 2013-06-14T09:30:29Z

src/addrman.h


-
+#include <cstdio>


What is this needed for?

The "printf"s on lines 406, 422, and 428.

printf is #define'd in util.h to be OutputDebugStringF.

Yeah, printf is our own macro.

sipa · 2013-06-14T09:42:05Z

I'm generally in favor of cleanups like this, and it indeed seems pure code movement + include changes.

However:

Some changes seem not up-to-date with recent refactors (see inline comments)
I'm not sure I like the increased use of forward declarations. They don't remove actual code dependencies, but they hide them. Of course we're already using them, and the compilation performance improvements it gives may be worth it. Other devs opinions?
Regarding uint64 and int64: I'd prefer including stdint.h and using int64_t and uint64_t instead, over redeclaring them all over the place.

@CodeShark Mind taking a look at this? Does it interfere with the dependencies cleanup?

laanwj · 2013-06-14T10:58:20Z

Getting rid of code dependencies is even better, but in the general case if it is possible do forward declarations instead of an #include in a header I'm in favor of that. It generates a flatter include hierarchy, let's not give the C++ compiler more work than need be.

Re: uint64_t/int32_t I agree. If available, they should be used, as they take the burden of determining data type sizes from us, at least on most platforms.

brandondahler · 2013-06-15T16:40:32Z

Addressed the specific problems references inline.

Hopefully should pass build, don't know why it built on my machine to be honest.

brandondahler · 2013-06-17T03:49:37Z

Delt with cstdio problems. Also normalized all [u]int64 types to [u]int64_t values from stdint.h, alongside replacing PRI64[xdu] with PRI[xdu]64 from inttypes.h .

luke-jr · 2013-06-17T03:52:01Z

@brandondahler Simply replacing [u]int64 types with stdint will probably break something - we tried this like a year ago and had to revert it :(

brandondahler · 2013-06-17T03:55:44Z

@luke-jr It was a non-negligable change, but it did compile and pass tests on my linux machine. Was the issue with compiling on other architectures?

luke-jr · 2013-07-16T21:46:52Z

@brandondahler Reviewing logs, it looks like it didn't build on 64-bit Fedora. I don't believe we ever fully diagnosed the reason, but:

[Wednesday, December 21, 2011] [3:15:25 PM] <jgarzik>   serialize.h: In function ‘unsigned int GetSerializeSize(int64
_t, int, int)’:
[Wednesday, December 21, 2011] [3:15:26 PM] <jgarzik>   serialize.h:139:21: error: redefinition of ‘unsigned int GetS
erializeSize(int64_t, int, int)’
[Wednesday, December 21, 2011] [3:15:26 PM] <jgarzik>   serialize.h:137:21: error: ‘unsigned int GetSerializeSize(lon
g int, int, int)’ previously defined here
[Wednesday, December 21, 2011] [3:15:38 PM] <jgarzik>   tree is full of broken
[Wednesday, December 21, 2011] [3:16:18 PM] Join        larsivi has joined this channel (~quassel@188.113.74.106).
[Wednesday, December 21, 2011] [3:16:25 PM] <jgarzik>   luke-jr: ^^
[Wednesday, December 21, 2011] [3:16:29 PM] <luke-jr>   jgarzik: works fine here…
[Wednesday, December 21, 2011] [3:16:56 PM] <jgarzik>   luke-jr: totally broke Fedora build (g++ 4.6.1)
[Wednesday, December 21, 2011] [3:17:05 PM] <jgarzik>   luke-jr: on 64-bit
[Wednesday, December 21, 2011] [3:17:32 PM] <jgarzik>   luke-jr: breakage is obvious.  64-bit platforms define int64_t==long int
[Wednesday, December 21, 2011] [3:17:49 PM] <luke-jr>   jgarzik: 4.5.3 here
[Wednesday, December 21, 2011] [3:18:31 PM] <jgarzik>   'int foo(int64_t)' is the same as 'int foo(long)'

luke-jr · 2013-07-22T20:09:10Z

src/compat.h

 #include <mswsock.h>
+#include <winsock2.h>


winsock2.h must be included before mswsock.h or build fails.

I also get a MinGW compiler warning that winsock2.h should be included before windows.h from this file in current master.

d:\mingw\mingw32\i686-w64-mingw32\include\winsock2.h:15: Warnung:#warning Please include winsock2.h before windows.h [-Wcpp] #warning Please include winsock2.h before windows.h ^

sipa · 2013-07-29T00:18:20Z

I think most other refactors that were in the pipeline are merged now. Care to rebase this?

brandondahler · 2013-07-29T03:41:57Z

Rebased branch onto master.

Diapolo · 2013-07-29T06:52:02Z

src/qt/addressbookpage.cpp

 #include <QMenu>
+#include <QSortFilterProxyModel>


Is this needed, as we include it in addressbookpage.h?

It is not included in addressbookpage.h--there is only forward declare (see below).

QT_BEGIN_NAMESPACE ... class QSortFilterProxyModel; ... QT_END_NAMESPACE

My fault, I didn't look correctly, sorry :).

No problems at all, thanks for reviewing the changes :)

jgarzik · 2013-08-25T03:11:14Z

It would be nice to collapse the commits, and merge in the [u]int64_t change first.

Looks pretty good overall, though I do worry there is a subtle compiler detail being missed in the int64/int64_t type changes.

Very much like seeing use of stdint.h and int64_t, rather than using our own type.

Diapolo · 2013-08-29T13:28:59Z

I also like the intention of this pull, can you rebase and perhaps follow @jgarzik so we can differentiate the type-changes and the cleanup changes :).

Edit: Also a merge-speedup is possible, if you create a Qt-only and a core-only pull, as @laanwj is able to merge Qt-pulls much faster than core changes.

Edit 2: @laanwj Any objection, to start using quint64 or qint64 in Qt code? Perhaps that could also be a scope of a separate Qt pull.

Diapolo · 2013-08-29T13:32:06Z

src/qt/bitcoin.cpp

@@ -2,29 +2,36 @@
 * W.J. van der Laan 2011-2012
 */

+


Nit: remove this new-line.

Diapolo · 2013-09-07T10:56:43Z

I really hope to see PullTester happy and some final ACKs for your great work here.

brandondahler · 2013-09-08T04:47:28Z

Rebased and squashed small commits (made sure that the resuting rebased version is equivalent to the previously merged version).

I plan on going over the source again to re-alphabetize etc.

gavinandresen · 2013-09-09T01:30:26Z

Pull-tester is fixed, so if it is complaining there is something wrong with your pull. See the test.log to debug.

sipa · 2013-11-10T02:07:07Z

src/main.h


 class CBlock;
+class CBlockHeader;


No need to forward declare this; CBlockHeader is defined in core, which is included.

brandondahler · 2013-11-10T05:32:19Z

Address @sipa comments. I think most of the main.h problems were caused by rebasing/merging so many times.

sipa · 2013-11-10T09:30:29Z

src/main.cpp

@@ -17,6 +16,15 @@
 #include "txdb.h"
 #include "txmempool.h"
 #include "ui_interface.h"
+#include "util.h"
+#include "wallet.h"


Since #3115, this shouldn't be needed anymore.

Oh, this is my fault: it was supposed to remove setpwalletRegistered and its critical section, but apparently didn't. In any case, they're obsolete now, and that can remove the dependency on CWallet.

sipa · 2013-11-10T09:48:30Z

ACK, apart from my nits above.

625188f Cleanup code using forward declarations. (Brandon Dahler)

sipa · 2013-11-10T11:22:13Z

Some suggested changes, which seem to work: sipa/bitcoin@0762f57

Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.

brandondahler · 2013-11-10T15:38:05Z

Merged sipa/bitcoin@header-cleanup (sipa/bitcoin@0762f57)

BitcoinPullTester · 2013-11-10T15:53:35Z

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/51ed9ec971614aebdbfbd9527aba365dd0afd437 for binaries and test log.
This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/
Contact BlueMatt on freenode if something looks broken.

sipa · 2013-11-10T18:19:14Z

ACK. Thanks a lot for the effort of writing this, and maintaining it.

51ed9ec Cleanup code using forward declarations. (Brandon Dahler)

Rewrite `if (var = func())` in a less confusing way

sipa reviewed Jun 14, 2013
View reviewed changes

sipa mentioned this pull request Jun 14, 2013

Refactor chain-specific tweaks into a CChainParams class and introduce a regtest mode #2632

Merged

luke-jr reviewed Jul 22, 2013
View reviewed changes

Diapolo reviewed Jul 29, 2013
View reviewed changes

Diapolo reviewed Aug 29, 2013
View reviewed changes

src/qt/bitcoin.cpp

@@ -2,29 +2,36 @@

* W.J. van der Laan 2011-2012

*/

Copy link

Diapolo Aug 29, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: remove this new-line.

sipa reviewed Nov 10, 2013
View reviewed changes

src/main.h

class CBlock;

class CBlockHeader;

Copy link

Member

sipa Nov 10, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to forward declare this; CBlockHeader is defined in core, which is included.

sipa reviewed Nov 10, 2013
View reviewed changes

sipa mentioned this pull request Nov 10, 2013

Cleanup refactoring of coins/mempool #3199

Merged

sipa added a commit to sipa/bitcoin that referenced this pull request Nov 10, 2013

Merge pull request bitcoin#2767

b643ce4

625188f Cleanup code using forward declarations. (Brandon Dahler)

sipa added a commit that referenced this pull request Nov 10, 2013

Merge pull request #2767

f76c122

51ed9ec Cleanup code using forward declarations. (Brandon Dahler)

sipa merged commit 51ed9ec into bitcoin:master Nov 10, 2013

brandondahler deleted the header-cleanup branch November 11, 2013 01:22

Diapolo mentioned this pull request Nov 11, 2013

massive compiler warning spam after #2767 #3236

Closed

Sporklin mentioned this pull request Mar 17, 2015

Fix thread name setting dogecoin/dogecoin#1063

Merged

sipa mentioned this pull request Apr 12, 2018

doc: Refine header include policy #12933

Merged

Bushstar pushed a commit to Bushstar/omnicore that referenced this pull request Apr 5, 2019

Slightly refactor ProcessInstantSendLock (bitcoin#2767)

6350adf

Rewrite `if (var = func())` in a less confusing way

maflcko mentioned this pull request Jun 6, 2019

refactor: Remove unused includes #16129

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Cleanup code by using forward declarations and other methods... #2767

Cleanup code by using forward declarations and other methods... #2767

Uh oh!

Conversation

brandondahler commented Jun 14, 2013

Theory

Quantitative Results

Method

Order

Uh oh!

gmaxwell commented Jun 14, 2013

Uh oh!

brandondahler commented Jun 14, 2013

Uh oh!

laanwj commented Jun 14, 2013

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sipa commented Jun 14, 2013

Uh oh!

laanwj commented Jun 14, 2013

Uh oh!

brandondahler commented Jun 15, 2013

Uh oh!

brandondahler commented Jun 17, 2013

Uh oh!

luke-jr commented Jun 17, 2013

Uh oh!

brandondahler commented Jun 17, 2013

Uh oh!

luke-jr commented Jul 16, 2013

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sipa commented Jul 29, 2013

Uh oh!

brandondahler commented Jul 29, 2013

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgarzik commented Aug 25, 2013

Uh oh!

Diapolo commented Aug 29, 2013

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Diapolo commented Sep 7, 2013

Uh oh!

brandondahler commented Sep 8, 2013

Uh oh!

gavinandresen commented Sep 9, 2013

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brandondahler commented Nov 10, 2013

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sipa commented Nov 10, 2013

Uh oh!

sipa commented Nov 10, 2013

Uh oh!

brandondahler commented Nov 10, 2013

Uh oh!