Avoid permanent cs_main/cs_wallet lock during RescanFromTime #11281

jonasschnelli · 2017-09-07T23:40:12Z

Right now, we are holding cs_main/cs_wallet during the whole rescan process (which can take a couple of hours).
This was probably only done because of laziness and it is an important show-stopper for #11200 (GUI rescan abort).

jonasschnelli · 2017-09-07T23:40:46Z

Best reviewed with ?w=1 (https://github.com/bitcoin/bitcoin/pull/11281/files?w=1)

practicalswift · 2017-09-08T07:26:09Z

@jonasschnelli Which variables are guarded by cs_main and cs_wallet in these specific cases?

I'll double-check against my lock annotations to make sure they are in sync.

promag · 2017-09-11T13:35:22Z

Awesome, will review.

sipa · 2017-09-12T00:01:47Z

src/wallet/wallet.cpp

-    LogPrintf("%s: Rescanning last %i blocks\n", __func__, startBlock ? chainActive.Height() - startBlock->nHeight + 1 : 0);
+    CBlockIndex* startBlock = nullptr;
+    {
+        LOCK2(cs_main, cs_wallet);


What needs cs_wallet here?

sipa · 2017-09-12T00:07:36Z

src/wallet/wallet.cpp

+        double dProgressStart = 0;
+        double dProgressTip = 0;
+        {
+            LOCK2(cs_main, cs_wallet);


What needs cs_wallet here?

sipa · 2017-09-12T00:11:31Z

src/wallet/wallet.cpp

@@ -1582,13 +1588,17 @@ CBlockIndex* CWallet::ScanForWalletTransactions(CBlockIndex* pindexStart, bool f

            CBlock block;
            if (ReadBlockFromDisk(block, pindex, Params().GetConsensus())) {
+                LOCK2(cs_main, cs_wallet);


What needs cs_main here?

chainActive.

sipa · 2017-09-12T00:11:43Z

src/wallet/wallet.cpp

                for (size_t posInBlock = 0; posInBlock < block.vtx.size(); ++posInBlock) {
                    AddToWalletIfInvolvingMe(block.vtx[posInBlock], pindex, posInBlock, fUpdate);
                }
            } else {
                ret = pindex;
            }
-            pindex = chainActive.Next(pindex);
+            {
+                LOCK2(cs_main, cs_wallet);


What needs cs_wallet here?

jonasschnelli · 2017-09-16T03:53:43Z

Updated and reduced locks after @sipa's mentioning.

achow101 · 2017-09-18T15:49:43Z

utACK 1d367effa8e7a19303ae46a6b871084cbaaf5a0a

promag · 2017-09-18T21:46:42Z

src/wallet/wallet.cpp

        fAbortRescan = false;
        fScanningWallet = true;

        ShowProgress(_("Rescanning..."), 0); // show rescan progress in GUI as dialog or on splashscreen, if -rescan on startup
-        double dProgressStart = GuessVerificationProgress(chainParams.TxData(), pindex);
-        double dProgressTip = GuessVerificationProgress(chainParams.TxData(), chainActive.Tip());
+        double dProgressStart = 0;


Only chainActive.Tip() needs the lock? So keep old code and:

CBlockIndex* tip = nullptr; { LOCK(cs_main); tip = chainActive.Tip(); } double dProgressStart = GuessVerificationProgress(chainParams.TxData(), pindex); double dProgressTip = GuessVerificationProgress(chainParams.TxData(), tip);

BTW, this patch allows the tip to be update so dProgressTip should be inside the loop?

theuni · 2017-09-19T22:00:18Z

src/wallet/wallet.cpp

-            pindex = chainActive.Next(pindex);
+            {
+                LOCK(cs_main);
+                pindex = chainActive.Next(pindex);


What if there's a reorg between iterations?

Does anything weird happen if we (for ex): begin scanning, receive a new best block with our tx from p2p, then re-scan here?

Seems to me that we should take a copy of ChainActive, traverse, compare to progress to chainActive.Tip(), and repeat until they match. Something like (just a quick sketch, probably broken):

CBlockIndex* pindex = pindexStart; CChain chain; while (!fAbortRescan) { { LOCK(cs_main); if (pindex == chainActive.Tip()) { break; } chain.SetTip(chainActive.Tip()); } if (pindex) { // start where we left off. const CBlockIndex* fork = chain.FindFork(pindex); if (fork != pindex) { // Need to undo the orphans. } pindex = chain.Next(fork); } while(!fAbortRescan && pindex) { /* do stuff ... pindex = chain.Next(pindex); */ } pindex = chain.Tip(); }

That almost entirely avoids locking cs_main, while ensuring that we're never jumping chains.

It could also be done with just the pindex and GetAncestor(), to skip the memory overhead of the second chain copy.

I'm not sure if this is really a problem. During rescans, IMO it won't hurt to scan an orphaned block nor does it hurt if you do not scan up to the tip in case you have connected new blocks during the time of re-scanning. This, because those new blocks must also have been scanned by the wallet logic (outside of ScanForWalletTransactions).

If a reorg happens during rescan, we want to continue rescanning using the current chain, not the outdated one.

This, because those new blocks must also have been scanned by the wallet logic (outside of ScanForWalletTransactions).

Indeed.
... I think it works out like this. Though @theuni does put a good point.

Maybe break the rescan as soon as possible if the above tip is not in the current chain? Then add here:

if (!chainActive.Contains(tip)) break;

In case of a reorg, wouldn't chainActive.Next(pindex); return a nullptr leading to stop the scan, which should be totally fine. During the reorg, the blocks of the new chain must have been scanned by the wallet via ConnectBlock.

Suppose the following:

A B C D E F G H I J - chain to rescan, tip = J

(A) B C D E F G H I J - scanning block A

A (B) C D E F G H I J - scanning block B

A (B) C D E F G H K L M - scanning block B, reorg after H

Should only rescan to H inclusive? Agree it's an edge case.

BTW, if the tip changes dProgressTip could be updated?

Yea, I think there is a real issue here. I (believe) it could be fixed by doing the AddToWalletIfInvolvingMe inside a cs_main (which is going to be at least mostly required since ReadBlockFromDisk requires cs_main) with a chainActive.Contains() check before doing the AddToWalletIfInvolvingMe. That should put us back to the original state of anything we add to the wallet is on the main chain and, thus, will either be Disconnected then Connected in the callbacks, only Connected, or not included in a callback at all.

I still don't fully understand the concerns (please help me).

If there is a reorg, all blocks of the new chain will be scanned by the wallet via ConnectBlock (no need to rescan again, we only need to rescan the stuff that doesn't come in via Connect/Disconnect block).

If we scan blocks that are no longer in the main-chain, and we could find a transaction in those blocks that is relevant to our wallet, so be it. The wallet does display such transactions as unconfirmed... same as we would have a reorg outside of the rescan process.

Heh, sorry, I didnt fully specify my concern because I thought the above discussion did, but I dont think the above comments are correct. As you point out, I do not believe there is a race where you simply miss transactions. However, there is a race where you may end up marking a transaction as in the wrong block:

A -> B -> ....
-> C -> ...

Both B and C have the transaction you're interested in.

You go to load block B from a rescan, but it has already been reorg'd and the notificatitons for disconnect(B) and connect(C) have already fired. Thus, you have the transaction marked as in block C, which is correct, but then you finish rescanning and mark the transaction as in block B, which now makes you think the transaction has been conflicted and has negative GetDepthInMainChain.

As I mentioned, this is most simply fixed by adding a "LOCK(cs_main); if (!chainActive.Contains(pindex)) break;" outside the AddToWalletIfInvolvingMe loop.

promag · 2017-09-19T23:54:05Z

Sorry for the bad reference ☝️ ...

jonasschnelli · 2017-09-22T03:49:26Z

Fixed @promag's code folding/style issue

jonasschnelli · 2017-11-15T20:53:58Z

Reviewers welcome (to make progress with GUI rescan abort #11200), best reviewed with ?w=1

laanwj · 2017-11-16T11:35:11Z

~~utACK 726fe69~~. There might be a slight overhead to locking for every block, likely only relevant for the first blocks of the chain which have few transactions. In any case this is preferable to the current situation.

(removed my utACK for now, this does not get reorgs right in case the order in which blocks are scanned matters, which I'm not sure about)

promag

LGTM, will play a bit.

promag · 2017-11-27T15:03:26Z

src/wallet/wallet.cpp

-            pindex = chainActive.Next(pindex);
+            {
+                LOCK(cs_main);
+                pindex = chainActive.Next(pindex);


Maybe break the rescan as soon as possible if the above tip is not in the current chain? Then add here:

if (!chainActive.Contains(tip)) break;

promag · 2017-11-27T15:12:02Z

BTW, see question #11281 (comment).

promag · 2017-11-27T15:33:31Z

This also allows concurrent rescans correct? It will mess up the progress dialog no?

laanwj · 2017-11-28T08:51:19Z

This also allows concurrent rescans correct? It will mess up the progress dialog no?

Concurrent rescans on multiple wallets would be nice, multiple concurrent rescans on one wallet would be dangerous or at least counter-productive (so we might want to track that?).
Progress dialog issues can be fixed later.

jonasschnelli · 2017-11-29T20:08:43Z

Rebased the PR.

Added checks to make sure only one rescan per time and per wallet can be executed (with appropriate error responses)
Extended the lock reduction to the new rescanblockchain call

jonasschnelli · 2017-11-29T20:12:05Z

Best reviewed with w=1.
Easy way to test multiple parallel rescans by adding a sleep within the inner rescan loop.

TheBlueMatt · 2017-12-04T15:01:54Z

src/wallet/wallet.cpp

        double dProgressStart = GuessVerificationProgress(chainParams.TxData(), pindex);
-        double dProgressTip = GuessVerificationProgress(chainParams.TxData(), chainActive.Tip());
+        double dProgressTip = GuessVerificationProgress(chainParams.TxData(), tip);


GuessVerificationProcess needs cs_main (probably best to lock cs_main in GuessVerificationProcess).

TheBlueMatt · 2017-12-04T15:11:32Z

src/wallet/wallet.cpp

@@ -1669,6 +1674,7 @@ CBlockIndex* CWallet::ScanForWalletTransactions(CBlockIndex* pindexStart, CBlock

            CBlock block;
            if (ReadBlockFromDisk(block, pindex, Params().GetConsensus())) {
+                LOCK(cs_wallet);


(one line up) ReadBlockFromDisk in CBlockIndex* form requires cs_main.

TheBlueMatt · 2017-12-04T15:32:23Z

src/wallet/wallet.cpp

-            pindex = chainActive.Next(pindex);
+            {
+                LOCK(cs_main);
+                pindex = chainActive.Next(pindex);


Yea, I think there is a real issue here. I (believe) it could be fixed by doing the AddToWalletIfInvolvingMe inside a cs_main (which is going to be at least mostly required since ReadBlockFromDisk requires cs_main) with a chainActive.Contains() check before doing the AddToWalletIfInvolvingMe. That should put us back to the original state of anything we add to the wallet is on the main chain and, thus, will either be Disconnected then Connected in the callbacks, only Connected, or not included in a callback at all.

TheBlueMatt · 2017-12-04T15:40:38Z

src/wallet/rpcdump.cpp

@@ -428,6 +432,10 @@ UniValue importpubkey(const JSONRPCRequest& request)
    if (fRescan && fPruneMode)
        throw JSONRPCError(RPC_WALLET_ERROR, "Rescan is disabled in pruned mode");

+    if (fRescan && pwallet->IsScanning()) {


This is very race-y. You could check check this, have another thread call rescan, and then end up with two parallell rescans.

Add a unit test for LockDirectory, introduced in bitcoin#11281.

Add a unit test for LockDirectory, introduced in #11281. Github-Pull: #12422 Rebased-From: 1d4cbd2 Tree-SHA512: 8186f4b22c65153e30ec1e0b68be20fd59d34e1fe565d99f3b5a302780125953b867fb14c37144de3fd7baf5751df94bf3901eebbb7b820491ca452706d4e205

Add a unit test for LockDirectory, introduced in bitcoin#11281.

90ba2df Fix missing cs_main lock for GuessVerificationProgress() (Jonas Schnelli) Pull request description: `GuessVerificationProgress()` needs `cs_main` due to accessing the `pindex->nChainTx`. This adds a `AssertLockHeld` in `GuessVerificationProgress()` and adds the missing locks in... * `LoadChainTip()` * `ScanForWalletTransactions()` (got missed in #11281) * GUI, `ClientModel::getVerificationProgress()` <--- **this may have GUI performance impacts**, but could be relaxed later with a cache or something more efficient. Tree-SHA512: 13302946571422375f32af8e396b9d2c1180f2693ea363aeba9e98c8266ddec64fe7862bfdcbb5a93a4b12165a61eec1e51e4e7d7a8515fa50879095dc163412

Add a unit test for LockDirectory, introduced in bitcoin#11281. Github-Pull: bitcoin#12422 Rebased-From: 1d4cbd2 Tree-SHA512: 8186f4b22c65153e30ec1e0b68be20fd59d34e1fe565d99f3b5a302780125953b867fb14c37144de3fd7baf5751df94bf3901eebbb7b820491ca452706d4e205

Return more information about scan status from ScanForWalletTransactions and try to describe the return value more clearly. There is a slight change in behavior here where rescans that end early due to a reorg will no longer trigger errors. (I incorrectly suggested that that they should trigger errors in the PR where this behavior was introduced: bitcoin#11281 (comment))

…scanFromTime 7f81250 Mention that other RPC calls report keys as "imported" while txns are still missing (Jonas Schnelli) ccd8ef6 Reduce cs_main lock in ReadBlockFromDisk, only read GetBlockPos under the lock (Jonas Schnelli) bc356b4 Make sure WalletRescanReserver has successfully reserved the rescan (Jonas Schnelli) dbf8556 Add RAII wallet rescan reserver (Jonas Schnelli) 8d0b610 Avoid pemanent cs_main/cs_wallet lock during wallet rescans (Jonas Schnelli) Pull request description: Right now, we are holding `cs_main`/`cs_wallet` during the whole rescan process (which can take a couple of hours). This was probably only done because of laziness and it is an important show-stopper for bitcoin#11200 (GUI rescan abort). Tree-SHA512: 0fc3f82d0ee9b2f013e6bacba8d59f7334306660cd676cd64c47bb305c4cb7c7a36219d6a6f76023b74e5fe87f3ab9fc7fd2439e939f71aef653fddb0a1e23b1 Signed-off-by: Pasta <pasta@dashboost.org> # Conflicts: # src/wallet/rpcdump.cpp # src/wallet/test/wallet_tests.cpp # src/wallet/wallet.h

…escanFromTime (#3411) * Merge bitcoin#11281: Avoid permanent cs_main/cs_wallet lock during RescanFromTime 7f81250 Mention that other RPC calls report keys as "imported" while txns are still missing (Jonas Schnelli) ccd8ef6 Reduce cs_main lock in ReadBlockFromDisk, only read GetBlockPos under the lock (Jonas Schnelli) bc356b4 Make sure WalletRescanReserver has successfully reserved the rescan (Jonas Schnelli) dbf8556 Add RAII wallet rescan reserver (Jonas Schnelli) 8d0b610 Avoid pemanent cs_main/cs_wallet lock during wallet rescans (Jonas Schnelli) Pull request description: Right now, we are holding `cs_main`/`cs_wallet` during the whole rescan process (which can take a couple of hours). This was probably only done because of laziness and it is an important show-stopper for bitcoin#11200 (GUI rescan abort). Tree-SHA512: 0fc3f82d0ee9b2f013e6bacba8d59f7334306660cd676cd64c47bb305c4cb7c7a36219d6a6f76023b74e5fe87f3ab9fc7fd2439e939f71aef653fddb0a1e23b1 Signed-off-by: Pasta <pasta@dashboost.org> # Conflicts: # src/wallet/rpcdump.cpp # src/wallet/test/wallet_tests.cpp # src/wallet/wallet.h * remove "LearnAllRelatedScripts" Signed-off-by: Pasta <pasta@dashboost.org> * fix importelectrumwallet Signed-off-by: Pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com>

…gress() 90ba2df Fix missing cs_main lock for GuessVerificationProgress() (Jonas Schnelli) Pull request description: `GuessVerificationProgress()` needs `cs_main` due to accessing the `pindex->nChainTx`. This adds a `AssertLockHeld` in `GuessVerificationProgress()` and adds the missing locks in... * `LoadChainTip()` * `ScanForWalletTransactions()` (got missed in bitcoin#11281) * GUI, `ClientModel::getVerificationProgress()` <--- **this may have GUI performance impacts**, but could be relaxed later with a cache or something more efficient. Tree-SHA512: 13302946571422375f32af8e396b9d2c1180f2693ea363aeba9e98c8266ddec64fe7862bfdcbb5a93a4b12165a61eec1e51e4e7d7a8515fa50879095dc163412

In bitcoin#11281, commit ccd8ef6 the `LOCK(cs_main)` scope was reduced for the wallet. This does the same for the rest API, which is a huge performance boost for `rest_block` call when used from multiple threads. My test setup, on an Intel i7 with 6 cores (12 threads), locked to 3.2GHz: 1. start a fully synced bitcoind, with this `bitcoin.conf`: ``` server=1 rest=1 rpcport=8332 rpcthreads=12 rpcworkqueue=64 txindex=1 dbcache=2000 ``` 2. Wait until log message shows `loadblk thread exit`, so that bitcoind is idle. 3. Run ApacheBench: 10000 requests, 12 parallel threads, fetching block nr. 600000 in binary: ``` ab -n 10000 -c 12 "http://127.0.0.1:8332/rest/block/00000000000000000007316856900e76b4f7a9139cfbfba89842c8d196cd5f91.bin" ``` Requests per second: * 97.434 [ms] (mean) on master * 20.431 [ms] this branch So this can process about 5 times as many requests, and saturates all my cores instead of keeping them idle waiting in the lock.

…ng RescanFromTime 1cde8b4 [Trivial] Fix some comment/output: 'bitcoind' --> 'pivxd' (random-zebra) 29c267e Mention that other RPC calls report keys as "imported" while txns are still missing (Jonas Schnelli) f773203 Reduce cs_main lock in ReadBlockFromDisk, only read GetBlockPos under the lock (Jonas Schnelli) 3e01665 Make sure WalletRescanReserver has successfully reserved the rescan (random-zebra) 4d05672 Add RAII wallet rescan reserver (Jonas Schnelli) c3fc22d Avoid pemanent cs_main/cs_wallet lock during wallet rescans (random-zebra) 69a7288 Handle TIMESTAMP_WINDOW within CWallet::RescanFromTime (Russell Yanofsky) fb12767 Make CWallet::RescanFromTime comment less ambiguous (Russell Yanofsky) 2d84bee Add RescanFromTime method and use from rpcdump (Russell Yanofsky) e8dfb32 Move birthday optimization out of ScanForWalletTransactions (random-zebra) Pull request description: Based on top of: - [x] #2279 Backports bitcoin#10412, and bitcoin#11281 ACKs for top commit: furszy: ACK 1cde8b4 and merging Tree-SHA512: 1c76ca9b7925299df796f72e171d70d1df9765d3b8d5976d7d280dd318474911cd8f173f686e57422a6f77fef145e4a9aae480b293e6b77ca4f93750688e997c

…escanFromTime (dashpay#3411) * Merge bitcoin#11281: Avoid permanent cs_main/cs_wallet lock during RescanFromTime 7f81250 Mention that other RPC calls report keys as "imported" while txns are still missing (Jonas Schnelli) ccd8ef6 Reduce cs_main lock in ReadBlockFromDisk, only read GetBlockPos under the lock (Jonas Schnelli) bc356b4 Make sure WalletRescanReserver has successfully reserved the rescan (Jonas Schnelli) dbf8556 Add RAII wallet rescan reserver (Jonas Schnelli) 8d0b610 Avoid pemanent cs_main/cs_wallet lock during wallet rescans (Jonas Schnelli) Pull request description: Right now, we are holding `cs_main`/`cs_wallet` during the whole rescan process (which can take a couple of hours). This was probably only done because of laziness and it is an important show-stopper for bitcoin#11200 (GUI rescan abort). Tree-SHA512: 0fc3f82d0ee9b2f013e6bacba8d59f7334306660cd676cd64c47bb305c4cb7c7a36219d6a6f76023b74e5fe87f3ab9fc7fd2439e939f71aef653fddb0a1e23b1 Signed-off-by: Pasta <pasta@dashboost.org> # Conflicts: # src/wallet/rpcdump.cpp # src/wallet/test/wallet_tests.cpp # src/wallet/wallet.h * remove "LearnAllRelatedScripts" Signed-off-by: Pasta <pasta@dashboost.org> * fix importelectrumwallet Signed-off-by: Pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com>

jonasschnelli added the Refactoring label Sep 7, 2017

laanwj added the Wallet label Sep 7, 2017

sipa reviewed Sep 12, 2017

View reviewed changes

jonasschnelli force-pushed the 2017/09/rescan_locks branch from 4c1f082 to 1d367ef Compare September 16, 2017 03:48

promag reviewed Sep 18, 2017

View reviewed changes

promag mentioned this pull request Sep 19, 2017

Remove redundant LOCK(…) and AssertLockHeld(…) #11361

Closed

theuni reviewed Sep 19, 2017

View reviewed changes

promag mentioned this pull request Sep 19, 2017

[test] Add getblockchaininfo functional test #11370

Merged

jonasschnelli force-pushed the 2017/09/rescan_locks branch from 1d367ef to 4bc57dc Compare September 22, 2017 03:47

jonasschnelli force-pushed the 2017/09/rescan_locks branch from 4bc57dc to 726fe69 Compare October 14, 2017 05:25

jonasschnelli mentioned this pull request Oct 29, 2017

Allow for aborting rescans in the GUI #11200

Merged

laanwj changed the title ~~Avoid pemanent cs_main/cs_wallet lock during RescanFromTime~~ Avoid permanent cs_main/cs_wallet lock during RescanFromTime Nov 16, 2017

promag reviewed Nov 27, 2017

View reviewed changes

jonasschnelli force-pushed the 2017/09/rescan_locks branch from 726fe69 to 0621022 Compare November 29, 2017 20:06

jonasschnelli force-pushed the 2017/09/rescan_locks branch from 0621022 to 78049d9 Compare November 29, 2017 20:10

TheBlueMatt reviewed Dec 4, 2017

View reviewed changes

laanwj added a commit to laanwj/bitcoin that referenced this pull request Feb 15, 2018

test: Add unit test for LockDirectory

1d4cbd2

Add a unit test for LockDirectory, introduced in bitcoin#11281.

Willtech pushed a commit to Willtech/bitcoin that referenced this pull request Feb 23, 2018

test: Add unit test for LockDirectory

0a4ec00

Add a unit test for LockDirectory, introduced in bitcoin#11281.

ryanofsky mentioned this pull request Nov 14, 2018

Remove uses of chainActive and mapBlockIndex in wallet code #14711

Merged

maflcko mentioned this pull request Oct 16, 2019

Importing privkey blocks p2p network traffic, results in connection resets and delayed TX relay #9201

Closed

martinus mentioned this pull request Jan 25, 2021

rpc: reduce LOCK(cs_min) scope in rest_block: ~5 times as many requests per second #21006

Closed

random-zebra mentioned this pull request Apr 1, 2021

[Wallet] Improve rescan API + avoid permanent locks during RescanFromTime PIVX-Project/PIVX#2281

Merged

1 task

jonatack mentioned this pull request Sep 5, 2021

consensus: don't call GetBlockPos in ReadBlockFromDisk without cs_main lock #22895

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Avoid permanent cs_main/cs_wallet lock during RescanFromTime #11281

Avoid permanent cs_main/cs_wallet lock during RescanFromTime #11281

Uh oh!

Conversation

jonasschnelli commented Sep 7, 2017

Uh oh!

jonasschnelli commented Sep 7, 2017

Uh oh!

practicalswift commented Sep 8, 2017

Uh oh!

promag commented Sep 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonasschnelli commented Sep 16, 2017

Uh oh!

achow101 commented Sep 18, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laanwj Nov 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

promag commented Sep 19, 2017

Uh oh!

jonasschnelli commented Sep 22, 2017

Uh oh!

jonasschnelli commented Nov 15, 2017

Uh oh!

laanwj commented Nov 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

promag left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

promag commented Nov 27, 2017

Uh oh!

promag commented Nov 27, 2017

Uh oh!

laanwj commented Nov 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonasschnelli commented Nov 29, 2017

Uh oh!

jonasschnelli commented Nov 29, 2017

Uh oh!

Choose a reason for hiding this comment

laanwj Nov 16, 2017 •

edited

Loading

laanwj commented Nov 16, 2017 •

edited

Loading

laanwj commented Nov 28, 2017 •

edited

Loading