Remove cap for dynamic config callback pool #7723
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
prathyushpv
reviewed
May 7, 2025
| @@ -18,7 +18,7 @@ var ( | |||
| "dynamicconfig.subscriptionCallback", | |||
| subscriptionCallbackSettings{ | |||
| MinWorkers: 10, | |||
| MaxWorkers: 100, | |||
| MaxWorkers: 1e9, // effectively unlimited | |||
There was a problem hiding this comment.
Then why don't we remove this config MaxWorkers altogether?
If we don't specify a maximum number of threads, is there value in using AdaptivePool? Can't we just create a new goroutine each time?
There was a problem hiding this comment.
Mainly: because the pool currently requires some max and I wanted to make a minimal change right now.
There's still some value in AdaptivePool: because of the timeout, in cases where callbacks are fast and don't call back into dynamic config (which is the common case), it won't create that many goroutines. Only when they call back into dynamic config and get stuck, then it'll create more goroutines than it otherwise would.
But yeah, you could argue the whole thing is overkill and we could just create new goroutines. It was an attempt to reduce the impact if we need to call something like 1M callbacks at once.
prathyushpv
approved these changes
May 8, 2025
ychebotarev
pushed a commit
that referenced
this pull request
May 8, 2025
## What changed? Set MaxWorkers for dynamic config subscription callback to 1e9 so there's no effective limit. ## Why? A batch of subscription callbacks are enqueued (though not called) while holding a lock. If a callback somehow makes a call back into dynamic config subscriptions, then it would have to wait for the batch to finish enqueueing. If there's more than MaxWorkers of those, that would cause a deadlock. We don't need to impose a limit on workers here. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s)
josesa
added a commit
to josesa/temporal
that referenced
this pull request
May 12, 2025
* main: (22 commits) Add host health metrics gauge (temporalio#7728) add rule expiration check (temporalio#7749) Add activity options to the pending activity info (temporalio#7727) Enable DLQ V2 for replication (temporalio#7746) chore: be smarter about when to use Stringer vs String (temporalio#7743) versioning entity workflows: enabling auto-restart pt1 (temporalio#7715) Refactor code generators (temporalio#7734) allow passive to generate replication tasks (temporalio#7713) Validate links in completion callbacks (temporalio#7726) CHASM: Engine Update/ReadComponent implementation (temporalio#7696) Enable transition history in dev env and tests (temporalio#7737) chore: Add Stringer tags (temporalio#7738) Add internal pod health check to DeepHealthCheck (temporalio#7709) Rename internal CHASM task processing interface (temporalio#7730) [Frontend] Log slow gRPC requests (temporalio#7718) Remove cap for dynamic config callback pool (temporalio#7723) Refactor updateworkflowoptions package (temporalio#7725) Remove a bunch of redundant utf-8 validation (temporalio#7720) [CHASM] Pure task processing - GetPureTasks, ExecutePureTasks (temporalio#7701) Send ActivityReset flag to the worker in heartbeat response (temporalio#7677) ...
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
Set MaxWorkers for dynamic config subscription callback to 1e9 so there's no effective limit.
Why?
A batch of subscription callbacks are enqueued (though not called) while holding a lock. If a callback somehow makes a call back into dynamic config subscriptions, then it would have to wait for the batch to finish enqueueing. If there's more than MaxWorkers of those, that would cause a deadlock. We don't need to impose a limit on workers here.
How did you test it?