Skip to content

Conversation

damian-orzechowski
Copy link
Contributor

@damian-orzechowski damian-orzechowski commented Nov 6, 2024

Changes

At the end of state ranges sync, when only large storage trie is left to be synced, snap batches will be processed sequentially. This change splits the range left to be requested for a given storage trie, if the processed data is less than 50% of requested range (maybe this shouldn't be 50%). The aim is to speed up processing of large storage tries, especially towards the end of snap ranges phase, when progress gets "stuck" at 100%. This is to aid the same problem as #7688, but with a slightly different approach.

Tested on OP-Mainnet - snap sync time:

  • Master: ~1:42 minutes (1h "stuck" at 100%)
  • After change: ~1:11 minutes (28min stuck at 93%)
Chain Master PR
op-mainnet 1 102 min 71 min
op-mainnet 2 138 min 90 min
op-mainnet 3 180 min 60 min
mainnet 1 71 min 66 min
mainnet 2 69 min 65 min

Types of changes

What types of changes does your code introduce?

  • Bugfix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a change that causes existing functionality not to work as expected)
  • Optimization
  • Refactoring
  • Documentation update
  • Build-related changes
  • Other: Description

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Notes on testing

Optional. Remove if not applicable.

Documentation

Requires documentation update

  • Yes
  • No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable.

Requires explanation in Release Notes

  • Yes
  • No

@damian-orzechowski damian-orzechowski marked this pull request as ready for review November 8, 2024 16:59
Copy link
Member

@LukaszRozmej LukaszRozmej left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please wait for @asdacap review too

@@ -29,6 +29,7 @@ public class ProgressTracker : IDisposable
public const int HIGH_STORAGE_QUEUE_SIZE = STORAGE_BATCH_SIZE * 100;
private const int CODES_BATCH_SIZE = 1_000;
public const int HIGH_CODES_QUEUE_SIZE = CODES_BATCH_SIZE * 5;
private const uint StorageRangeSplitFactor = 2;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you experimented with more radical splitting factors? 4? 8? 16? 32?

@LukaszRozmej LukaszRozmej requested a review from asdacap November 8, 2024 19:29
@asdacap
Copy link
Contributor

asdacap commented Nov 8, 2024

Can add unit test?

@damian-orzechowski damian-orzechowski requested review from rubo and a team as code owners November 11, 2024 09:13
@damian-orzechowski damian-orzechowski force-pushed the feature/snap-sync-parallel-storage-2 branch from 525c8aa to f734316 Compare November 11, 2024 10:31
@damian-orzechowski damian-orzechowski removed request for a team and rubo November 11, 2024 13:44
@damian-orzechowski
Copy link
Contributor Author

Can add unit test?

Added

Copy link
Contributor

@Scooletz Scooletz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Provided some comments. This seems similar to Paprika's ref counting.

I don't know how often the lock concurrent dictionary is used, but if it is not contended we could replace the concurrent with a lock (potentially, RWLSlim) and use some global atomicity. It might be the case that due to construction of the snap sync, we know that once the account is processed, it will never appear again. If this is the case it can be leveraged and used to know when to remove a thing from the dictionary (current condition).

In Paprika's case, read-only transactions use just a lock and a simple dictionary. Nothing fancy, but due to the fact that each tx is ref-counted, cannot be resurrected, once it's 0, it takes lock on the dictionary and removes itself.

{
lock (_lock)
{
Counter += value;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The counter seems to be not correlated at all with the ExecuteSafe method. I'd replace lock(_lock) with Interlocked.Add for both, Increment and Decrement methods. Unless in Decrement you want to be sure that no ExecuteSafe is happening at the moment

@@ -57,6 +59,8 @@ public class ProgressTracker : IDisposable
private ConcurrentQueue<ValueHash256> CodesToRetrieve { get; set; } = new();
private ConcurrentQueue<AccountWithStorageStartingHash> AccountsToRefresh { get; set; } = new();

private readonly ConcurrentDictionary<ValueHash256, IStorageRangeLock> _storageRangeLocks = new();
private readonly StorageRangeLockPassThrough _emptyStorageLock = new();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is the one that allows to pass on all the actions and is treated as the default. Some comment would be nice I believe.

Comment on lines 525 to 526
if (!_storageRangeLocks.TryGetValue(accountPath, out IStorageRangeLock lockInfo)) return;
lockInfo.Increment(number);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What @LukaszRozmej said above. What if you call IncrementStorageRangeLockIfExists you get no lockInfo and immediately it's followed by IncrementStorageRangeLock that does the update? Does it mean that something is broken here?

public interface IStorageRangeLock
{
IStorageRangeLock Increment(uint value = 1);
void Decrement(ValueHash256 key, bool removeFromOwner);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why key is needed here? Should it not be captured in the ctor of the lock, so that we just Decrement(bool removeFromOwner). You get the lock by key so the key should not be needed here I think.


public interface IStorageRangeLock
{
IStorageRangeLock Increment(uint value = 1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why increment returns the lock object? My feeling is that it's to satisfy the ConcurrentDictionary. I'd keep it simple and make it void.

}

return result;
}
finally
{
rangeLock?.Decrement(pathWithAccount.Path, canRemoveFurtherLock);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to take care of whether a lock should be removed or not? Should it not be the case that when the counter hits 0, we always remove?

void ExecuteSafe(Action action);
}

public class StorageRangeLock(uint counter, ConcurrentDictionary<ValueHash256, IStorageRangeLock> owner)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just use a sharded lock to avoid such complications.
Also, you can just use _pivot.UpdatedStorages.Add(pathWithAccount.Path); to the splitted storage so that healing will fix it. But it limits the number of storage that can be splitted though.

@Scooletz Scooletz mentioned this pull request Dec 5, 2024
Comment on lines 147 to 150
// This will work if all StorageRange requests share the same AccountWithPath object which seems to be the case.
// If this is not true, StorageRange request should be extended with a lock object.
// That lock object should be shared between all other StorageRange requests for same account.
lock(account.Account)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main change here!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main point of having this complexity around locking was to lock only when necessary - that is when split is done and there us a chance of actually having parallel processing of storage ranges for same account. If the overhead is negligible, I'm all for it - it's the simplest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On having a conversation with @Scooletz yesterday, there could be a way to optimize it easily, if there is significant overhead. Stitching only happens when there are boundary proof nodes - so can lock only if that happens.

Copy link
Member

@LukaszRozmej LukaszRozmej Dec 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lock is super cheap compared to I/O so locking even when not necessary shouldn't degrade performance. And you still paid for ConcurrentDictionary in your solution, so it might be comparable

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fine. That said, this code path is run quite a lot, might wannna double check if the lock is significant.

Copy link
Member

@LukaszRozmej LukaszRozmej Dec 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no contention then lock is very fast, if there is contention, well we still have to lock. Keep in mind in the previous solution we locked on potential contention while also accessed ConcurrentDictionary at least one time even with no contention. https://stackoverflow.com/a/72479230

@damian-orzechowski
Copy link
Contributor Author

damian-orzechowski commented Dec 6, 2024

This test run shows this may bring missing nodes on storage - cannot merge yet
https://github.com/NethermindEth/nethermind/actions/runs/12183618666

Looks like this has been caused by and error in the previous locking scheme. After changes from @LukaszRozmej it looks fine and state passes verification after sync:
https://github.com/NethermindEth/nethermind/actions/runs/12238172700

// This will work if all StorageRange requests share the same AccountWithPath object which seems to be the case.
// If this is not true, StorageRange request should be extended with a lock object.
// That lock object should be shared between all other StorageRange requests for same account.
lock (account.Account)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the account based locking and using the object that might be elsewhere locked or used for different purposes. Locking should have specific scoping and should be reasoned about locally.

Another take is that this lock is needed, correct me @damian-orzechowski if I'm wrong, only if we know that there's no pending requests. We could extract the check from StitchBoundaries and assert it explicitly before the lock.

Copy link
Member

@LukaszRozmej LukaszRozmej Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could create artificial object and pass it around, but I don't really see the point other than increasing the memory pressure and complicating the code. We added comments! :)

Done is probably better than perfect here.
If we want to refactor is to have parent request directly available when child request is being created. So the connection is obvious and not hidden in multiple method calls.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done some refactoring here #7930. The problem is that passing around the whole StorageRange request means also passing an index of the actual account processed (accounts are processed one at a time), so it doesn't look very nice.

Co-authored-by: lukasz.rozmej <lukasz.rozmej@gmail.com>
@damian-orzechowski
Copy link
Contributor Author

@Scooletz Scooletz marked this pull request as draft January 3, 2025 12:50
@Scooletz
Copy link
Contributor

Scooletz commented Jan 3, 2025

@damian-orzechowski Converted this PR to a draft for now.

@damian-orzechowski damian-orzechowski force-pushed the feature/snap-sync-parallel-storage-2 branch from c31f55b to f44de49 Compare January 28, 2025 09:52
@Scooletz
Copy link
Contributor

Not sure if ready for a review. If yes, re-request pls.

@damian-orzechowski damian-orzechowski marked this pull request as ready for review February 14, 2025 12:11
@damian-orzechowski damian-orzechowski merged commit b9ed459 into master Feb 20, 2025
80 checks passed
@damian-orzechowski damian-orzechowski deleted the feature/snap-sync-parallel-storage-2 branch February 20, 2025 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants