[WIP] 1.8.3 supermajority fix #6

patricklodder · 2015-08-07T14:24:10Z

Please don't merge this yet, needs ~~implementation in 1.10 AND~~ a regression test

This PR enables the SuperMajority logic again after it's been disabled since v1.5 and proposes a solution for rejecting version1 blocks without having to retroactively hardfork the params at 1.8 fork:

We cannot enforce version 1 block rejection on the live chains with a v2 supermajority because previous clients will disagree and possibly we do not have compatibility with the existing chain. I propose that instead, we piggyback the version 3 supermajority switch to also enforce the version 2 softfork and all is good. This is the best solution I could think of in terms of code impact and backward compatibility until a v3 supermajority is reached, but I'm extremely open to suggestions.

I kept the commits split as one enables the logic and one fixes the bug with v2 softforks.

rnicoll · 2015-08-07T14:55:36Z

Do check, but I think this is what I've done for 1.10 as well. Sorry, at work, catching stuff while things compile/boot

patricklodder · 2015-08-07T15:50:59Z

Took me some time to find why you fixed it like you did (just perform the check on every v >= 2 block rather than enforce a supermajority at the coinbase check) but it's actually better than the implementation here and I'd expect the result to be the same indeed.

I'll have to write a regtest for this after I fix the qa/rpc-tests hashing implementation to scrypt (is currently sha256d)

langerhans · 2015-08-07T18:43:47Z

So, it does indeed need a test, especially since the stuff differs between 1.8 and 1.10. I'm okay either way but would probably prefer the more strict on by @rnicoll (as I've just learned).

patricklodder · 2015-08-09T13:11:34Z

@langerhans: Morphed this to strict block version constraint enforcement with 14ca107

However, for consistency we should do the same with v3 constraints in 1.10 then, eg: change this implementation to match patricklodder@14ca107#diff-7ec3c68a81efff79b6ca22ac1f1eabbaR1970

patricklodder · 2015-08-09T14:07:37Z

Ok i just thought this through again and strict checks is a risk.

Currently, any block version higher than 1 will be accepted in any client without any constraint checks. If we were to do strict checks then we risk a single miner doing block.nVersion = 0xff and we will have a problem enforcing the next 252 block versions this way. So better to do the conditional enforcement as it would only become a problem then if the majority of the network decides to mine with version 0xff.

Propose to revert 14ca107 and instead implement SuperMajority v2 checking in 1.10... thoughts?

rnicoll · 2015-08-09T14:14:48Z

Unless I'm missing something, v2 supermajority breaks consensus (there's a v1 block past the point where we have 950 out of 1000 v2 blocks).

Actually not too sure of the problem 0xff gives us?

patricklodder · 2015-08-09T14:24:09Z

The problem is that I can make a v2 block now that will not honor v2 constraints and all 1.8.x clients will accept it. Then when we enforce v2 strictly in 1.10, there will be no consensus from my block forward.

rnicoll · 2015-08-09T14:26:23Z

Ah.... yeah, that's why the rules only enable past a supermajority (750/1000 or otherwise), so unless we're 51% attacked no single miner can achieve that. Can we have coinbase height checks enabled on a v3 supermajority instead of v2?

patricklodder · 2015-08-09T14:28:28Z

yes, that's the best way. I'll write a PR on 1.10 too

rnicoll · 2015-08-09T14:29:35Z

Awesome, I was going to do that but entirely approve of you doing things so I don't have to :-D

langerhans · 2015-08-09T15:07:47Z

I'll just nod and smile. No, really, I don't want us to put in anything that risks a divergence in consensus. So I'm good if we use the less strict way.

patricklodder · 2015-08-10T08:49:28Z

Reworked this and backported 1.10 time-based constraint values

rnicoll · 2015-08-16T19:09:43Z

I get the following error trying to run the unit tests, looks like a consensus issue has snuck in:

jrn@debian:~/dogecoin$ ./src/test/test_dogecoin
Running 126 test cases...
unknown location(0): fatal error in "CreateNewBlock_validity": memory access violation at address: 0x00000008: no mapping at fault address
miner_tests.cpp(86): last checkpoint

*** 1 failure detected in test suite "Bitcoin Test Suite"
test_dogecoin: /usr/include/boost/thread/pthread/recursive_mutex.hpp:101: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.
Aborted

This has been disabled since 1.5 for unclear reasons, but we need it for the bip66 softfork in 1.10 that is backported here. This commit re-enables the logic.

Because CBlockIndex::IsSuperMajority was disabled since 1.5, we cannot enforce version 1 block rejection on the live chains with a v2 supermajority. Instead, we piggyback the version 3 supermajority switch to also enforce the version 2 softfork and all is good.

Backport of dogecoin#1225 to 1.8.x, sets: - majority window to 2000 blocks on mainnet, 1000 on testnet - enforcement triggers to 1500 on mainnet, 501 on testnet - reject triggers to 1900 on mainnet, 750 on testnet

Since v2 and v1 blocks will be rejected at exactly the same time, only check for supermajority once to save 2000 iterations over blockheaders per block. Also moves the version check to AcceptBlockHeader() rather than AcceptBlock() to be able to reject early on.

patricklodder · 2015-09-09T13:50:50Z

cause of the test issue was a bad copy/paste in last commit. fixed and squashed it now.

[WIP] 1.8.3 supermajority fix

rnicoll · 2015-09-09T18:22:57Z

Thanks!

c521b3ac6 Merge #11: fixup define checks. Cleans up some oopses from #5. 8b1cd3753 fixup define checks. Cleans up some oopses from #5. 6b1508d6d Merge #6: Fixes typo fceb80542 Merge #10: Clean up compile-time warnings (gcc 7.1) 0ec2a343f Clean up compile-time warnings (gcc 7.1) d4c268a35 Merge #5: Move helper functions out of sse4.2 object 8d4eb0847 Add HasAcceleratedCRC32C to port_win.h 77cfbfd25 crc32: move helper functions out of port_posix_sse.cc 4c1e9e016 silence compiler warnings about uninitialized variables 495316485 Merge #2: Prefer std::atomic over MemoryBarrier 2953978ef Fixes typo f134284a1 Merge #1: Merge upstream LevelDB 1.20 ba8a445fd Prefer std::atomic over MemoryBarrier git-subtree-dir: src/leveldb git-subtree-split: c521b3ac654cfbe009c575eacf7e5a6e189bb5bb

6f53edb Acquire cs_main before ATMP call in block_assemble bench (James O'Beirne) Pull request description: Calling `bench_bitcoin` currently fails due to calling ATMP without acquiring cs_main first in the recently added block_assemble bench (bitcoin/bitcoin#13219). ``` $ cat <(uname -a) <(gcc --version) Linux james 4.4.0-119-generic dogecoin#143+jamesob SMP Mon Apr 16 21:47:24 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609 $ ./src/bench/bench_bitcoin WARNING: This is a debug build - may result in slower benchmarks. # Benchmark, evals, iterations, total, min, max, median Assertion failed: lock cs_main not held in validation.cpp:566; locks held: [1] 19323 abort (core dumped) ./src/bench/bench_bitcoin ``` ``` (gdb) bt #0 0x00007fbdc9cf5428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #1 0x00007fbdc9cf702a in __GI_abort () at abort.c:89 #2 0x0000555a19580dc5 in AssertLockHeldInternal (pszName=pszName@entry=0x555a19834549 "cs_main", pszFile=pszFile@entry=0x555a1988a001 "validation.cpp", nLine=nLine@entry=566, cs=cs@entry=0x555a19ba55c0 <cs_main>) at sync.cpp:157 #3 0x0000555a194b395f in AcceptToMemoryPoolWorker (chainparams=..., pool=..., state=..., ptx=std::shared_ptr (count 1, weak 0) 0x555a1bb819b0, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=1532964079, plTxnReplaced=0x0, bypass_limits=false, nAbsurdFee=@0x7ffcbc1719d8: 0, coins_to_uncache=std::vector of length 0, capacity 0, test_accept=false) at validation.cpp:566 #4 0x0000555a194ba661 in AcceptToMemoryPoolWithTime (chainparams=..., pool=..., state=..., tx=std::shared_ptr (count 1, weak 0) 0x555a1bb819b0, pfMissingInputs=pfMissingInputs@entry=0x0, nAcceptTime=<optimized out>, plTxnReplaced=0x0, bypass_limits=false, nAbsurdFee=0, test_accept=false) at validation.cpp:998 #5 0x0000555a194ba7ce in AcceptToMemoryPool (pool=..., state=..., tx=std::shared_ptr (count 1, weak 0) 0x555a1bb819b0, pfMissingInputs=pfMissingInputs@entry=0x0, plTxnReplaced=plTxnReplaced@entry=0x0, bypass_limits=bypass_limits@entry=false, nAbsurdFee=0, test_accept=false) at validation.cpp:1014 #6 0x0000555a19363fbe in AssembleBlock (state=...) at bench/block_assemble.cpp:102 #7 0x0000555a193654d3 in std::_Function_handler<void (benchmark::State&), void (*)(benchmark::State&)>::_M_invoke(std::_Any_data const&, benchmark::State&) (__functor=..., __args#0=...) at /usr/include/c++/5/functional:1871 #8 0x0000555a193501d7 in std::function<void (benchmark::State&)>::operator()(benchmark::State&) const (this=this@entry=0x555a1ba2cda0, __args#0=...) at /usr/include/c++/5/functional:2267 #9 0x0000555a1934ec4c in benchmark::BenchRunner::RunAll (printer=..., num_evals=5, scaling=<optimized out>, filter=..., is_list_only=false) at bench/bench.cpp:121 #10 0x0000555a1934ade9 in main (argc=<optimized out>, argv=<optimized out>) at bench/bench_bitcoin.cpp:92 ``` Tree-SHA512: fdd7b28ff123ccea7a4f334d53f735d0c0f94aa9cc52520c2dd34dca45d78c691af64efcd32366fc472fedffbd79591d2be2bb3bfc4a5186e8712b6b452d64e3

8963b2c qt: Improve comments in WalletController::getOrCreateWallet() (Hennadii Stepanov) 5fcfee6 qt: Call setParent() in the parent's context (Hennadii Stepanov) 5659e73 qt: Add ObjectInvoke template function (Hennadii Stepanov) Pull request description: The `setParent(parent)` internally calls `QCoreApplication::sendEvent(parent, QChildEvent)` that implies running in the thread which created the parent object. That is not the case always, and an internal assertion fails in the debug mode. Steps to reproduce this issue on master (007e15d) on Linux Mint 20 (x86_64): ``` $ make -C depends DEBUG=1 $ CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure $ make $ QT_FATAL_WARNINGS=1 lldb src/qt/bitcoin-qt -- --regtest -debug=qt (lldb) target create "src/qt/bitcoin-qt" Current executable set to '/home/hebasto/GitHub/bitcoin/src/qt/bitcoin-qt' (x86_64). (lldb) settings set -- target.run-args "--regtest" "-debug=qt" (lldb) run Process 431562 launched: '/home/hebasto/GitHub/bitcoin/src/qt/bitcoin-qt' (x86_64) # load wallet via GUI Process 431562 stopped * thread #24, name = 'QThread', stop reason = signal SIGABRT frame #0: 0x00007ffff794518b libc.so.6`__GI_raise(sig=2) at raise.c:51:1 (lldb) bt * thread #24, name = 'QThread', stop reason = signal SIGABRT * frame #0: 0x00007ffff794518b libc.so.6`__GI_raise(sig=2) at raise.c:51:1 frame #1: 0x00007ffff7924859 libc.so.6`__GI_abort at abort.c:79:7 frame #2: 0x0000555556508ec4 bitcoin-qt`::qt_message_fatal((null)=<unavailable>, context=<unavailable>, message=<unavailable>) at qlogging.cpp:1690:15 frame #3: 0x00005555565099cf bitcoin-qt`QMessageLogger::fatal(this=<unavailable>, msg=<unavailable>) const at qlogging.cpp:796:21 frame #4: 0x000055555650479d bitcoin-qt`qt_assert_x(where=<unavailable>, what=<unavailable>, file=<unavailable>, line=<unavailable>) at qglobal.cpp:3088:46 frame #5: 0x0000555556685733 bitcoin-qt`QCoreApplicationPrivate::checkReceiverThread(receiver=0x0000555557b27510) at qcoreapplication.cpp:557:5 frame #6: 0x00005555567ced86 bitcoin-qt`QApplication::notify(this=0x00007fffffffd4a0, receiver=0x0000555557b27510, e=0x00007fff9a7f8ce0) at qapplication.cpp:2956:27 frame #7: 0x0000555556685d31 bitcoin-qt`QCoreApplication::notifyInternal2(receiver=0x0000555557b27510, event=0x00007fff9a7f8ce0) at qcoreapplication.cpp:1024:24 frame #8: 0x00005555566c9224 bitcoin-qt`QObjectPrivate::setParent_helper(QObject*) [inlined] QCoreApplication::sendEvent(event=<unavailable>, receiver=<unavailable>) at qcoreapplication.h:233:59 frame #9: 0x00005555566c9210 bitcoin-qt`QObjectPrivate::setParent_helper(this=0x00007fff85855260, o=0x0000555557b27510) at qobject.cpp:2036 frame #10: 0x00005555566c9b41 bitcoin-qt`QObject::setParent(this=<unavailable>, parent=<unavailable>) at qobject.cpp:1980:24 frame #11: 0x0000555555710be8 bitcoin-qt`WalletController::getOrCreateWallet(std::unique_ptr<interfaces::Wallet, std::default_delete<interfaces::Wallet> >) + 2534 ... ``` Fixes #18835. ACKs for top commit: ryanofsky: Code review ACK 8963b2c. No changes since last review, just rebase because of conflict on some adjacent lines jonasschnelli: utACK 8963b2c Tree-SHA512: fef615904168717df3d8a0bd85eccc3eef990cc3e66c9fa280c8ef08ea009a7cb5a2a4f868ed0be3c0fe5bf683e8465850b5958deb896fdadd22d296186c9586

fa6c114 test: Add sanitizer suppressions for AMD EPYC CPUs (MarcoFalke) Pull request description: Currently the ci system only runs on intel cpus (and some arm devices), but it won't run on CPUs `Using the 'shani(1way,2way)' SHA256 implementation` (excerpt from debug log). For reference, google cloud CPUs (which is what Cirrus CI uses) print `Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation` The traceback I got: ``` crypto/sha256_shani.cpp:87:18: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'size_t' (aka 'unsigned long') #0 0x55c0000e95ec in sha256_shani::Transform(unsigned int*, unsigned char const*, unsigned long) /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/crypto/sha256_shani.cpp:87:18 #1 0x55bfffb926f8 in (anonymous namespace)::SelfTest() /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/crypto/sha256.cpp:517:9 #2 0x55bfffb906ed in SHA256AutoDetect[abi:cxx11]() /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/crypto/sha256.cpp:626:5 #3 0x55bfff87ab97 in BasicTestingSetup::BasicTestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/util/setup_common.cpp:104:5 #4 0x55bffe885877 in main /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/qt/test/test_main.cpp:52:27 #5 0x7f20c3bf60b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) #6 0x55bffe7a5f6d in _start (/root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/qt/test/test_bitcoin-qt+0x1d00f6d) SUMMARY: UndefinedBehaviorSanitizer: unsigned-integer-overflow crypto/sha256_shani.cpp:87:18 in ACKs for top commit: laanwj: Anyhow ACK fa6c114 Tree-SHA512: 968a1d28eedec58c337b1323862f583cb1bcd78c5f03396940b9ab53ded12f8c6652877909aba05ee5586532137418fd817ff979bd7bef6e07856094f9d7f9b1

Otherwise it is not possible to run bench_bitcoin with clang-12 + ASAN compiled. Output: $ src/bench/bench_bitcoin bench/nanobench.h:1107:15: runtime error: left shift of 4982565676696827473 by 27 places cannot be represented in type 'uint64_t' (aka 'unsigned long') #0 0x5623d6a13137 in ankerl::nanobench::Rng::rotl(unsigned long, unsigned int) /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/./bench/nanobench.h:1107:15 #1 0x5623d6a13137 in ankerl::nanobench::Rng::operator()() /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/./bench/nanobench.h:1075:10 #2 0x5623d6a05c5b in ankerl::nanobench::Rng::Rng(unsigned long) /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/./bench/nanobench.h:3135:9 #3 0x5623d6a0ca51 in ankerl::nanobench::detail::IterationLogic::Impl::Impl(ankerl::nanobench::Bench const&) /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/./bench/nanobench.h:2206:13 #4 0x5623d69f8f73 in ankerl::nanobench::detail::IterationLogic::IterationLogic(ankerl::nanobench::Bench const&) /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/./bench/nanobench.h:2215:18 #5 0x5623d690f165 in ankerl::nanobench::Bench& ankerl::nanobench::Bench::run<AddrManAdd(ankerl::nanobench::Bench&)::$_0>(AddrManAdd(ankerl::nanobench::Bench&)::$_0&&) /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/./bench/nanobench.h:1114:28 #6 0x5623d690e26e in AddrManAdd(ankerl::nanobench::Bench&) /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bench/addrman.cpp:76:11 #7 0x5623d69279d6 in void std::__invoke_impl<void, void (*&)(ankerl::nanobench::Bench&), ankerl::nanobench::Bench&>(std::__invoke_other, void (*&)(ankerl::nanobench::Bench&), ankerl::nanobench::Bench&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:60:14 #8 0x5623d6927921 in std::enable_if<is_invocable_r_v<void, void (*&)(ankerl::nanobench::Bench&), ankerl::nanobench::Bench&>, void>::type std::__invoke_r<void, void (*&)(ankerl::nanobench::Bench&), ankerl::nanobench::Bench&>(void (*&)(ankerl::nanobench::Bench&), ankerl::nanobench::Bench&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:110:2 #9 0x5623d692775f in std::_Function_handler<void (ankerl::nanobench::Bench&), void (*)(ankerl::nanobench::Bench&)>::_M_invoke(std::_Any_data const&, ankerl::nanobench::Bench&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/std_function.h:291:9 #10 0x5623d692dbd5 in std::function<void (ankerl::nanobench::Bench&)>::operator()(ankerl::nanobench::Bench&) const /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/std_function.h:622:14 #11 0x5623d692cd44 in benchmark::BenchRunner::RunAll(benchmark::Args const&) /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bench/bench.cpp:65:13 #12 0x5623d69282bf in main /bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bench/bench_bitcoin.cpp:63:5 #13 0x7f6812010564 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x28564) #14 0x5623d685f4dd in _start (/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/bench/bench_bitcoin+0x13754dd) SUMMARY: UndefinedBehaviorSanitizer: invalid-shift-base bench/nanobench.h:1107:15 in $ clang --version Ubuntu clang version 12.0.0-1ubuntu1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

patricklodder mentioned this pull request Aug 7, 2015

Apply BIP 66 changes to 1.8.3 client dogecoin/dogecoin#1209

Merged

patricklodder force-pushed the 1.8.3-supermajority-fix branch from 2ac0edc to b36bd55 Compare August 7, 2015 14:43

patricklodder mentioned this pull request Aug 10, 2015

Re-enable SuperMajority triggered activation for v2 block constraints dogecoin/dogecoin#1231

Merged

patricklodder force-pushed the 1.8.3-supermajority-fix branch from 14ca107 to 14c36f4 Compare August 10, 2015 08:48

patricklodder added 4 commits September 9, 2015 15:44

enable CBlockIndex::IsSuperMajority

f770e4e

This has been disabled since 1.5 for unclear reasons, but we need it for the bip66 softfork in 1.10 that is backported here. This commit re-enables the logic.

Use the same time-based constraints as in 1.10

7cdc820

Backport of dogecoin#1225 to 1.8.x, sets: - majority window to 2000 blocks on mainnet, 1000 on testnet - enforcement triggers to 1500 on mainnet, 501 on testnet - reject triggers to 1900 on mainnet, 750 on testnet

patricklodder force-pushed the 1.8.3-supermajority-fix branch from db3dd47 to a48cd64 Compare September 9, 2015 13:50

rnicoll pushed a commit that referenced this pull request Sep 9, 2015

Merge pull request #6 from patricklodder/1.8.3-supermajority-fix

0a59c7c

[WIP] 1.8.3 supermajority fix

rnicoll merged commit 0a59c7c into rnicoll:1.8.3-bip66 Sep 9, 2015

patricklodder deleted the 1.8.3-supermajority-fix branch September 11, 2015 14:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] 1.8.3 supermajority fix #6

[WIP] 1.8.3 supermajority fix #6

Uh oh!

patricklodder commented Aug 7, 2015

Uh oh!

rnicoll commented Aug 7, 2015

Uh oh!

patricklodder commented Aug 7, 2015

Uh oh!

langerhans commented Aug 7, 2015

Uh oh!

patricklodder commented Aug 9, 2015

Uh oh!

patricklodder commented Aug 9, 2015

Uh oh!

rnicoll commented Aug 9, 2015

Uh oh!

patricklodder commented Aug 9, 2015

Uh oh!

rnicoll commented Aug 9, 2015

Uh oh!

patricklodder commented Aug 9, 2015

Uh oh!

rnicoll commented Aug 9, 2015

Uh oh!

langerhans commented Aug 9, 2015

Uh oh!

patricklodder commented Aug 10, 2015

Uh oh!

rnicoll commented Aug 16, 2015

Uh oh!

patricklodder commented Sep 9, 2015

Uh oh!

rnicoll commented Sep 9, 2015

Uh oh!

Uh oh!

[WIP] 1.8.3 supermajority fix #6

[WIP] 1.8.3 supermajority fix #6

Uh oh!

Conversation

patricklodder commented Aug 7, 2015

Uh oh!

rnicoll commented Aug 7, 2015

Uh oh!

patricklodder commented Aug 7, 2015

Uh oh!

langerhans commented Aug 7, 2015

Uh oh!

patricklodder commented Aug 9, 2015

Uh oh!

patricklodder commented Aug 9, 2015

Uh oh!

rnicoll commented Aug 9, 2015

Uh oh!

patricklodder commented Aug 9, 2015

Uh oh!

rnicoll commented Aug 9, 2015

Uh oh!

patricklodder commented Aug 9, 2015

Uh oh!

rnicoll commented Aug 9, 2015

Uh oh!

langerhans commented Aug 9, 2015

Uh oh!

patricklodder commented Aug 10, 2015

Uh oh!

rnicoll commented Aug 16, 2015

Uh oh!

patricklodder commented Sep 9, 2015

Uh oh!

rnicoll commented Sep 9, 2015

Uh oh!

Uh oh!