Skip to content

[LogServer] Do not share record cache with log-server #3620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 4, 2025
Merged

[LogServer] Do not share record cache with log-server #3620

merged 2 commits into from
Aug 4, 2025

Conversation

AhmedSoliman
Copy link
Contributor

@AhmedSoliman AhmedSoliman commented Aug 2, 2025 •
edited
Loading

This addresses a major cache thrashing issue in big-ish clusters. Log servers would prematurely evict sequencer-local records when they receive less important records on log-server. Additionally, this significantly reduces contention on the cache which saves some CPU time. Tests show a ~4-8% throughput increase when CPU is saturated.

Stack created with Sapling. Best reviewed with ReviewStack.

This was referenced Aug 2, 2025
Merged
Merged
Merged
Merged
Merged
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2025 •
edited
Loading

Test Results

 5 files   -   2   5 suites   - 2   59s ⏱️ - 3m 59s
25 tests  -  29  25 ✅  -  28  0 💤  - 1  0 ❌ ±0 
34 runs   - 189  34 ✅  - 186  0 💤  - 3  0 ❌ ±0 

Results for commit f3e9ed3. ± Comparison against base commit 83e6966.

This pull request removes 54 and adds 25 tests. Note that renamed tests count towards both. 
dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[1]
dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[2]
dev.restate.sdktesting.tests.CallOrdering ‑ ordering(boolean[], Client)[3]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[1]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[2]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromAdminAPI(BlockingOperation, Client, URI)[3]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[1]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[2]
dev.restate.sdktesting.tests.Cancellation ‑ cancelFromContext(BlockingOperation, Client)[3]
dev.restate.sdktesting.tests.Combinators ‑ awakeableOrTimeoutUsingAwaitAny(Client)
…

dev.restate.sdktesting.tests.AwakeableIngressEndpointTest ‑ completeWithFailure(Client)
dev.restate.sdktesting.tests.AwakeableIngressEndpointTest ‑ completeWithSuccess(Client)
dev.restate.sdktesting.tests.BackCompatibilityTest$NewVersion ‑ createAwakeable(Client)
dev.restate.sdktesting.tests.BackCompatibilityTest$NewVersion ‑ startOneWayProxyCall(Client)
dev.restate.sdktesting.tests.BackCompatibilityTest$NewVersion ‑ startProxyCall(Client)
dev.restate.sdktesting.tests.BackCompatibilityTest$NewVersion ‑ startRetryableOperation(Client)
dev.restate.sdktesting.tests.BackCompatibilityTest$OldVersion ‑ completeAwakeable(Client)
dev.restate.sdktesting.tests.BackCompatibilityTest$OldVersion ‑ completeRetryableOperation(Client)
dev.restate.sdktesting.tests.BackCompatibilityTest$OldVersion ‑ proxyCallShouldBeDone(Client)
dev.restate.sdktesting.tests.BackCompatibilityTest$OldVersion ‑ proxyOneWayCallShouldBeDone(Client)
…

♻️ This comment has been updated with latest results.

@AhmedSoliman AhmedSoliman marked this pull request as ready for review August 4, 2025 09:41
@AhmedSoliman
[Fabric] Adds ability to disable network compression via config
862eaf1 
Introduces `networking.disable_compression` that disables Zstd compression for grpc services. This can be useful for cpu-challenged environments. Compression is still enabled by default.

```
// intentionally empty
```
@AhmedSoliman
[LogServer] Do not share record cache with log-server
f3e9ed3 
This addresses a major cache thrashing issue in big-ish clusters. Log servers would prematurely evict sequencer-local records when they receive less important records on log-server. Additionally, this significantly reduces contention on the cache which saves some CPU time. Tests show a ~4-8% throughput increase when CPU is saturated.
@AhmedSoliman AhmedSoliman changed the title [LogServer] do not share record cache with log-server [LogServer] Do not share record cache with log-server Aug 4, 2025
pcholakov
pcholakov approved these changes Aug 4, 2025
View reviewed changes
Copy link
Contributor

@pcholakov pcholakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Insightful, thanks for fixing this! I can see why one would think to put a record in the cache on the log server's write path.

@AhmedSoliman AhmedSoliman mentioned this pull request Aug 4, 2025
Merged
@AhmedSoliman AhmedSoliman merged commit f3e9ed3 into main Aug 4, 2025
53 checks passed
@AhmedSoliman AhmedSoliman deleted the pr3620 branch August 4, 2025 14:17
@github-actions github-actions bot locked and limited conversation to collaborators Aug 4, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@pcholakov pcholakov pcholakov approved these changes

@muhamadazmy muhamadazmy Awaiting requested review from muhamadazmy

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AhmedSoliman @pcholakov