-
Notifications
You must be signed in to change notification settings - Fork 549
Split storage ranges to parallelize execution #7733
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Split storage ranges to parallelize execution #7733
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please wait for @asdacap review too
src/Nethermind/Nethermind.Synchronization/SnapSync/ProgressTracker.cs
Outdated
Show resolved
Hide resolved
@@ -29,6 +29,7 @@ public class ProgressTracker : IDisposable | |||
public const int HIGH_STORAGE_QUEUE_SIZE = STORAGE_BATCH_SIZE * 100; | |||
private const int CODES_BATCH_SIZE = 1_000; | |||
public const int HIGH_CODES_QUEUE_SIZE = CODES_BATCH_SIZE * 5; | |||
private const uint StorageRangeSplitFactor = 2; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
have you experimented with more radical splitting factors? 4? 8? 16? 32?
src/Nethermind/Nethermind.Synchronization/SnapSync/ProgressTracker.cs
Outdated
Show resolved
Hide resolved
Can add unit test? |
525c8aa
to
f734316
Compare
Added |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Provided some comments. This seems similar to Paprika's ref counting.
I don't know how often the lock concurrent dictionary is used, but if it is not contended we could replace the concurrent with a lock (potentially, RWLSlim) and use some global atomicity. It might be the case that due to construction of the snap sync, we know that once the account is processed, it will never appear again. If this is the case it can be leveraged and used to know when to remove a thing from the dictionary (current condition).
In Paprika's case, read-only transactions use just a lock
and a simple dictionary. Nothing fancy, but due to the fact that each tx is ref-counted, cannot be resurrected, once it's 0, it takes lock on the dictionary and removes itself.
{ | ||
lock (_lock) | ||
{ | ||
Counter += value; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The counter seems to be not correlated at all with the ExecuteSafe
method. I'd replace lock(_lock)
with Interlocked.Add
for both, Increment
and Decrement
methods. Unless in Decrement
you want to be sure that no ExecuteSafe
is happening at the moment
@@ -57,6 +59,8 @@ public class ProgressTracker : IDisposable | |||
private ConcurrentQueue<ValueHash256> CodesToRetrieve { get; set; } = new(); | |||
private ConcurrentQueue<AccountWithStorageStartingHash> AccountsToRefresh { get; set; } = new(); | |||
|
|||
private readonly ConcurrentDictionary<ValueHash256, IStorageRangeLock> _storageRangeLocks = new(); | |||
private readonly StorageRangeLockPassThrough _emptyStorageLock = new(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this is the one that allows to pass on all the actions and is treated as the default. Some comment would be nice I believe.
if (!_storageRangeLocks.TryGetValue(accountPath, out IStorageRangeLock lockInfo)) return; | ||
lockInfo.Increment(number); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What @LukaszRozmej said above. What if you call IncrementStorageRangeLockIfExists
you get no lockInfo
and immediately it's followed by IncrementStorageRangeLock
that does the update? Does it mean that something is broken here?
public interface IStorageRangeLock | ||
{ | ||
IStorageRangeLock Increment(uint value = 1); | ||
void Decrement(ValueHash256 key, bool removeFromOwner); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why key
is needed here? Should it not be captured in the ctor
of the lock, so that we just Decrement(bool removeFromOwner)
. You get the lock by key so the key should not be needed here I think.
|
||
public interface IStorageRangeLock | ||
{ | ||
IStorageRangeLock Increment(uint value = 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why increment returns the lock object? My feeling is that it's to satisfy the ConcurrentDictionary
. I'd keep it simple and make it void.
} | ||
|
||
return result; | ||
} | ||
finally | ||
{ | ||
rangeLock?.Decrement(pathWithAccount.Path, canRemoveFurtherLock); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need to take care of whether a lock should be removed or not? Should it not be the case that when the counter hits 0, we always remove?
void ExecuteSafe(Action action); | ||
} | ||
|
||
public class StorageRangeLock(uint counter, ConcurrentDictionary<ValueHash256, IStorageRangeLock> owner) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd just use a sharded lock to avoid such complications.
Also, you can just use _pivot.UpdatedStorages.Add(pathWithAccount.Path);
to the splitted storage so that healing will fix it. But it limits the number of storage that can be splitted though.
// This will work if all StorageRange requests share the same AccountWithPath object which seems to be the case. | ||
// If this is not true, StorageRange request should be extended with a lock object. | ||
// That lock object should be shared between all other StorageRange requests for same account. | ||
lock(account.Account) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Main change here!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The main point of having this complexity around locking was to lock only when necessary - that is when split is done and there us a chance of actually having parallel processing of storage ranges for same account. If the overhead is negligible, I'm all for it - it's the simplest.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On having a conversation with @Scooletz yesterday, there could be a way to optimize it easily, if there is significant overhead. Stitching only happens when there are boundary proof nodes - so can lock only if that happens.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lock is super cheap compared to I/O so locking even when not necessary shouldn't degrade performance. And you still paid for ConcurrentDictionary in your solution, so it might be comparable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be fine. That said, this code path is run quite a lot, might wannna double check if the lock is significant.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there is no contention then lock is very fast, if there is contention, well we still have to lock. Keep in mind in the previous solution we locked on potential contention while also accessed ConcurrentDictionary at least one time even with no contention. https://stackoverflow.com/a/72479230
|
// This will work if all StorageRange requests share the same AccountWithPath object which seems to be the case. | ||
// If this is not true, StorageRange request should be extended with a lock object. | ||
// That lock object should be shared between all other StorageRange requests for same account. | ||
lock (account.Account) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like the account
based locking and using the object that might be elsewhere locked or used for different purposes. Locking should have specific scoping and should be reasoned about locally.
Another take is that this lock is needed, correct me @damian-orzechowski if I'm wrong, only if we know that there's no pending requests. We could extract the check from StitchBoundaries
and assert it explicitly before the lock.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we could create artificial object and pass it around, but I don't really see the point other than increasing the memory pressure and complicating the code. We added comments! :)
Done is probably better than perfect here.
If we want to refactor is to have parent request directly available when child request is being created. So the connection is obvious and not hidden in multiple method calls.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done some refactoring here #7930. The problem is that passing around the whole StorageRange
request means also passing an index of the actual account processed (accounts are processed one at a time), so it doesn't look very nice.
Co-authored-by: lukasz.rozmej <lukasz.rozmej@gmail.com>
More issues found - do not merge: |
@damian-orzechowski Converted this PR to a draft for now. |
c31f55b
to
f44de49
Compare
Not sure if ready for a review. If yes, re-request pls. |
Changes
At the end of state ranges sync, when only large storage trie is left to be synced, snap batches will be processed sequentially. This change splits the range left to be requested for a given storage trie, if the processed data is less than 50% of requested range (maybe this shouldn't be 50%). The aim is to speed up processing of large storage tries, especially towards the end of snap ranges phase, when progress gets "stuck" at 100%. This is to aid the same problem as #7688, but with a slightly different approach.
Tested on OP-Mainnet - snap sync time:
Types of changes
What types of changes does your code introduce?
Testing
Requires testing
If yes, did you write tests?
Notes on testing
Optional. Remove if not applicable.
Documentation
Requires documentation update
If yes, link the PR to the docs update or the issue with the details labeled
docs
. Remove if not applicable.Requires explanation in Release Notes