Use a dedicated thread pool for creating indexheader.Readers #2048
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3b435a8 to
e21e7f7
Compare
Merged
3 tasks
e21e7f7 to
75eead4
Compare
|
Marking this as ready for review since we're still in the process of testing #2019. If it turns out #2019 is sufficient to mitigate the issue, we can remove use of the threadpool entirely. However, there's no harm in adding this feature and leaving it disabled while gathering more information (and based on testing, using the threadpool for requests is not the correct solution). |
Remove use of a dedicated thread pool for store-gateway requests. Instead, only use the thread pool for creation of new indexheader.Reader instances since this is when most I/O on memory mapped files happens. Using dedicated threads for entire requests introduced a new way for store- gateways to fail when becoming overloaded. Since the threads in the pool are a constrained resource, when slow requests monopolize them all other requests are blocked waiting for a thread. By moving use of the thread pool to creation of indexheader.Reader instances, we minimize the amount of time that a thread must be used for while also making sure that we don't use the thread pool at all when not needed. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
75eead4 to
7fe6648
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
Remove use of a dedicated thread pool for store-gateway requests. Instead,
only use the thread pool for creation of new indexheader.Reader instances
since this is when most I/O on memory mapped files happens.
Using dedicated threads for entire requests introduced a new way for store-
gateways to fail when becoming overloaded. Since the threads in the pool are
a constrained resource, when slow requests monopolize them all other requests
are blocked waiting for a thread. By moving use of the thread pool to
creation of indexheader.Reader instances, we minimize the amount of time
that a thread must be used for while also making sure that we don't use
the thread pool at all when not needed.
Signed-off-by: Nick Pillitteri nick.pillitteri@grafana.com
Which issue(s) this PR fixes or relates to
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]