I find it scary, that a bug like this can be introduced. I am wondering if there are not any tools that check the code for references to uninitialized variables?

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

Fixes what? Both are already initialized..

Fixes what? Both are already initialized..

It's used for the computation of BIP9WarningHeight, and by that time it isn't initialized (I missed it first, too!).

I find it scary, that a bug like this can be introduced. I am wondering if there are not any tools that check the code for references to uninitialized variables?

The tests are being run in various sanitizers, and valgrind can do this IIRC. I'm surprised it wasn't caught.

ACK 6fcd798

Should fix #16697.

@promag It looks like consensus.nMinerConfirmationWindow was indeed being read uninitialised on L74 prior to the change suggested by this PR?

consensus.MinBIP9WarningHeight = consensus.SegwitHeight + consensus.nMinerConfirmationWindow;

This code was introduced in fdb3e8f in PR #16713 which was merged in to master on September 27, 2019.

Really fascinating that this 1.) wasn't caught automatically by our static and dynamic analysis, 2.) wasn't caught by our review process and 3.) was unnoticed in master for over a month. Room for improvement! :)

@bitcoinVBR

Thanks a lot for reporting this! What an excellent first time contribution! Hope to see more great contributions from you.

Sorry about the initial misunderstanding in your original PR #17433 :)

Really fascinating that this 1.) wasn't caught automatically by our static and dynamic analysis,

Port to rust when? 😄

Oh!

Worth noting is that the read of nMinerConfirmationWindow takes place on mainnet and testnet but not on regtest:

$ grep -E '(nMinerConfirmationWindow|CBaseChainParams|MinBIP9WarningHeight)' \
      src/chainparams.cpp | grep -A2 " = *CBaseChainParams"
        strNetworkID = CBaseChainParams::MAIN;
        consensus.MinBIP9WarningHeight = consensus.SegwitHeight + consensus.nMinerConfirmationWindow;
        consensus.nMinerConfirmationWindow = 2016; // nPowTargetTimespan / nPowTargetSpacing
        strNetworkID = CBaseChainParams::TESTNET;
        consensus.MinBIP9WarningHeight = consensus.SegwitHeight + consensus.nMinerConfirmationWindow;
        consensus.nMinerConfirmationWindow = 2016; // nPowTargetTimespan / nPowTargetSpacing
        strNetworkID =  CBaseChainParams::REGTEST;
        consensus.MinBIP9WarningHeight = 0;
        consensus.nMinerConfirmationWindow = 144; // Faster than normal for regtest (144 instead of 2016)

ACK 6fcd798
Really surprising that apparently no compiler spit out a warning about this uninitialized access...

Worth noting is that the read of nMinerConfirmationWindow takes place on mainnet and testnet but not on regtest:

Good catch. If there is no unit test that instantiates the main / testnet chain, that'd explain why sanitizer CI checks don't notice it. But it seems that quite a few do, for example:

pow_tests.cpp:    const auto consensus = CreateChainParams(CBaseChainParams::MAIN)->GetConsensus();

@laanwj I think there might be a more subtle issue here too - note that MinBIPWarningHeight appears to have the correct value (481824 + 2016 == 483840) when reaching Condition(…) on mainnet: #16713 (comment).

I think we might be chasing two issues here where the uninitialised read is the first one :)

Good catch. If there is no unit test that instantiates the main / testnet chain, that'd explain why sanitizer CI checks don't notice it. But it seems that quite a few do, for example:

We have functional tests that run on the testnet chain. So this should have been caught by a sanitizer (assuming one was enabled)

I don't think we have the memory sanitizer enabled anywhere

@bitcoinVBR

Thanks a lot for reporting this! What an excellent first time contribution! Hope to see more great contributions from you.

Sorry about the initial misunderstanding in your original PR #17433 :)

Thanks! I'm happy to help. I actually only found this bug because of my work on BitcoinV. I'll continue to keep an eye out for anything fishy.

Compiler writers get a lot of heat for exploiting UB to the fullest for optimisation purposes, but in this case it seems like we got really lucky -- they "fixed" our code (assuming Clang -O2) 😉

Look at the Clang -O2 case where we get the 483840 (481824 + 2016 == 483840) we intended (!) :)

$ clang++-8 -O0 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 481824
consensus.MinBIP9WarningHeight[2] == 4678000
consensus.MinBIP9WarningHeight[3] == 481824
$ clang++-8 -O1 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 586474128
consensus.MinBIP9WarningHeight[2] == 4678896
consensus.MinBIP9WarningHeight[3] == -1100057664
$ clang++-8 -O2 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 483840
consensus.MinBIP9WarningHeight[2] == 483840
consensus.MinBIP9WarningHeight[3] == 483840
$ g++-8 -O0 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 503884
consensus.MinBIP9WarningHeight[2] == 481824
consensus.MinBIP9WarningHeight[3] == 503884
$ g++-8 -O1 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 481824
consensus.MinBIP9WarningHeight[2] == 481824
consensus.MinBIP9WarningHeight[3] == 481824
$ g++-8 -O2 -o repro repro.cpp && ./repro
consensus.MinBIP9WarningHeight[1] == 481824
consensus.MinBIP9WarningHeight[2] == 481824
consensus.MinBIP9WarningHeight[3] == 481824
$ cat repro.cpp
#include <cstdint>
#include <iostream>

namespace Consensus {
struct Params {
  int SegwitHeight;
  int MinBIP9WarningHeight;
  uint32_t nMinerConfirmationWindow;
};
} // namespace Consensus

class CChainParams {
public:
  const Consensus::Params &GetConsensus() const { return consensus; }

protected:
  CChainParams() {}

  Consensus::Params consensus;
};

class CMainParams : public CChainParams {
public:
  CMainParams() {
    consensus.SegwitHeight = 481824;
    consensus.MinBIP9WarningHeight =
        consensus.SegwitHeight + consensus.nMinerConfirmationWindow;
    consensus.nMinerConfirmationWindow = 2016;
  }
};

int main() {
  CMainParams main_params_1;
  std::cout << "consensus.MinBIP9WarningHeight[1] == "
            << main_params_1.GetConsensus().MinBIP9WarningHeight << std::endl;

  CMainParams main_params_2;
  std::cout << "consensus.MinBIP9WarningHeight[2] == "
            << main_params_2.GetConsensus().MinBIP9WarningHeight << std::endl;

  CMainParams main_params_3;
  std::cout << "consensus.MinBIP9WarningHeight[3] == "
            << main_params_3.GetConsensus().MinBIP9WarningHeight << std::endl;
}

:)

@jonatack The output you posted from the testing of PR #16713 in #16713 (comment) ...

2019-09-20T11:26:36Z SegwitHeight 481824
2019-09-20T11:26:36Z nMinerConfirmationWindow 2016
2019-09-20T11:26:36Z MinBIPWarningHeight 483840
2019-09-20T11:26:36Z pindex->nHeight 164836
2019-09-20T11:26:36Z pindex->nVersion 1
2019-09-20T11:26:36Z ComputeBlockVersion 536870912

... could it be the case that you happened to test with Clang with the project default optimisation level -O2? :) If so, could you retry using GCC (any optimisation level), or Clang with -O0 or -O1 and see if you get the same results?

@practicalswift I tried gcc 9.2.1 and it happened to optimize out the bug for me as well. Though running the same binary in valgrind still jells at me, so I can't make any sense out of that.

@MarcoFalke What is the output of ./repro for binaries compiled at the different optimisation levels?

could it be the case that you happened to test with Clang with the project default optimisation level -O2? :) If so, could you retry using GCC (any optimisation level), or Clang with -O0 or -O1 and see if you get the same results?

It was with gcc version 8.3.0 (Debian 8.3.0-6) with no optimisation flags.

It was with gcc version 8.3.0 (Debian 8.3.0-6) with no optimisation flags.

The default is O2 for Bitcoin Core

I'd prefer to just hard code the MinBIP9WarningHeight, eg:

        consensus.MinBIP9WarningHeight = 483840; // segwit activation height + miner confirmation window

Hard-coding the value means that even if the member initializations are moved around again, nMinerConfirmationWindow will not be accessed before initialization.

MinBIP9WarningHeight is currently the only member variable that is set from the value of a different member, except the hashGenesisBlock which is checked with an assert against a hard-coded value.

I was about to suggest that the members should be marked const, so that initialization and initialization order is enforced by the compiler. Though, I believe that makes the code less readable as C++ does not have named parameters that can be passed into a constructor.

@bitcoinVBR Could you please adjust the patch as suggested by @jnewbery and then squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

Are there any bounty rewards for these finds?

The Bitcoin Core project does not have any official bounty program AFAIK, but the is nothing stopping individuals from donating to other individuals who they think have done valuable security research.

(Aside: My personal view is that we could do a much better job recognising good security research: especially the non-glamorous kind of low-level security research. As an example: if we had a proper incentive structure in place I guess someone would be running bitcoind binaries compiled from master with -O{0,1,2} running under valgrind around the clock just waiting to report an issue like this. Incentives matter. Unfortunately I don't know the optimal incentive structure (the mix between public recognition vs official bug bounty vs donations vs just being polite and thanking researchers vs long-term funding, etc.), but I'm fairly certain we're not at the global optimum right now :))

Shameless plug: People interested in running bitcoind under valgrind might want to help out by reviewing #17455 :)

Maybe you can get funding for such a thing from one of the exchanges or companies in the space, that's always how developer funding has worked, but the 'bitcoin core' project is a loose name for people contributing to an open source project, and does not manage funds, and should not manage funds, and will never have an "official incentive structure".

[…] the 'bitcoin core' project is a loose name for people contributing to an open source project, and does not manage funds, and should not manage funds, and will never have an "official incentive structure".

Agree 100% if we are talking monetary reward.

Note that incentive structure does not necessarily imply monetary reward: giving proper credit and treating researchers nicely goes a surprisingly long way when it comes to attracting security research :)

To bring this on-topic again (sorry, my fault!): I hope we can make this a good example by making sure @bitcoinVBR is properly credited for allowing us to ship 0.19.0 without an "optimisation unstable" UpdateTip(…) (due to the uninitialised read) :)

As all contributors to the release, they've been added to the authors list in 0.19.0.1 (as NullFunctor, as that's the name on their git mail address, @bitcoinVBR: if you want to be credited under a different name let me know) and this PR has been added to the changelog.

This is not a security issue so I'm not sure why you bring that up. It doesn't matter either, bugs are bugs. All people that contribute deserve credit.

Backported to 0.19 in 6ec0dc1.

ACK on the rebase 6ec0dc1.

consensus.MinBIP9WarningHeight = consensus.SegwitHeight + consensus.nMinerConfirmationWindow;

Really fascinating that this 1.) wasn't caught automatically by our static and dynamic analysis,

Looks like -Werror=uninitialized is able to catch this, but only if the initialisation is done in a Consensus::Params constructor via a member initializer list rather than a code block (and -O3 is specified if using g++). (I thought I was getting g++ to catch it in code blocks as well, but can't duplicate that now)

@ajtowns Worth mentioning from a dynamic analysis perspective is that both Valgrind and MemorySanitizer (-fsanitize=memory) catch this too :)