Set SCHED_BATCH priority on the loadblk thread. #12618

eklitzke · 2018-03-06T18:46:45Z

Today I came across #10271, and while reading the discussion #6358 was linked to. Linux systems have a SCHED_BATCH scheduler priority that is useful for threads like loadblk. You can find the full details at sched(7), but I'll quote the relevant part of the man page below:

...this policy will cause the scheduler to always assume that the thread is
CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty
with respect to wakeup behavior, so that this thread is mildly disfavored in
scheduling decisions.

This policy is useful for workloads that are noninteractive, but do not want to
lower their nice value, and for workloads that want a deterministic scheduling
policy without interactivity causing extra preemptions (between the workload's
tasks).

I think this change is useful independently of #10271 and irrespective of whether that change is merged. Under normal operation the loadblk thread will just import mempool.dat. However, if Bitcoin is started with -reindex or -reindex-chainstate this thread will use a great deal of CPU while it rebuilds the chainstate database (and the block database in the case of -reindex). By setting SCHED_BATCH this thread is less likely to interfere with interactive tasks (e.g. the user's web browser, text editor, etc.).

I'm leaving the nice value unchanged (which also affects scheduling decisions) because I think that's better set by the user. Likewise I'm not using ioprio_set(2) because it can cause the thread to become completely I/O starved (and knowledgeable users can use ionice(1) anyway).

laanwj · 2018-03-06T19:03:19Z

src/util.cpp

@@ -955,3 +955,10 @@ fs::path AbsPathForConfigVal(const fs::path& path, bool net_specific)
 {
    return fs::absolute(path, GetDataDir(net_specific));
 }
+
+void ScheduleBatchPriority(pid_t tid) {


pid_t is not available on windows, I think it would be better to remove the parameter as it's always passed as 0.

I think you're right.

laanwj · 2018-03-06T19:03:39Z

Is loadblk thread the only thread this is relevant for? what about the script verification threads?

promag · 2018-03-06T19:48:18Z

Is this the windows equivalent?

SetThreadPriority(GetCurrentThread(), THREAD_MODE_BACKGROUND_BEGIN)

eklitzke · 2018-03-06T20:34:47Z

On Linux SCHED_BATCH just prevents batch tasks from pre-empting normal tasks (i.e. normal tasks will run until their timeslice expires, or another normal tasks pre-empts them). When picking the next task to run the scheduler considers SCHED_OTHER (i.e. normal) and SCHED_BATCH tasks equally. This is why setting SCHED_BATCH tasks are only "mildly disfavored", because they'll still be considered to run using the normal metrics whenever the scheduler ticks (usually 100Hz or 1000Hz).

It sounds like Windows THREAD_MODE_BACKGROUND_BEGIN is much more extreme. It looks like maybe the Windows equivalent is THREAD_PRIORITY_BELOW_NORMAL but I do not know a lot about Windows.

I don't know as much about the script verification threads. If we are going to set SCHED_BATCH for loadblk then it makes sense to me to set the script verification threads to SCHED_BATCH as well, at least until the loadblk thread completes. You can freely switch back from SCHED_BATCH to SCHED_OTHER so it would be possible to only run them in the batch mode while reindexing is happening, or we could just always have them set at batch priority.

JeremyRubin · 2018-03-06T23:00:16Z

Concept ACK for this:

one note: you would want something realtime for scriptcheck threads while they are validating a block incoming from network -- we want these to take priority over other threads aggressively.

eklitzke · 2018-03-09T18:42:53Z

Re: Jeremy's comment, there are a few ways to increase the priority of the script verification thread to ensure low-latency block relay. On Linux there exists SCHED_FIFO and SCHED_RR threads which give threads semi-realtime scheduling guarantees, and can ensure that such threads are always prioritized above other threads. Linux also supports per-thread nice values (this is a Linux extension, POSIX specifies that nice values are per-process).

Using the other scheduling modes is better for realtime behavior, but somewhat dangerous because such threads can pre-empt all other non-realtime processes, meaning that this could be used as a DOS vector. Changing the nice value is a bit safer. I will probably look into this as part of another PR because I'd like to benchmark that change.

maflcko · 2018-03-18T23:14:56Z

I think the commits can be squashed?

eklitzke · 2018-03-19T06:58:08Z

Thanks @MarcoFalke , squashed to one commit. I'm interested into looking at the RT change Jeremy suggested as well, but that's more impactful so I'd rather do that separately unless people want to see that change done here.

ryanofsky

utACK a2ed276b52aec97d37257b7aad73f7f332a926d3

ryanofsky · 2018-03-22T14:20:45Z

src/init.cpp

@@ -627,6 +627,7 @@ void ThreadImport(std::vector<fs::path> vImportFiles)
 {
    const CChainParams& chainparams = Params();
    RenameThread("bitcoin-loadblk");
+    ScheduleBatchPriority();


Would be nice to log the error message if sched_batch returns -1. EINVAL / EPERM cases would be good to know about.

ryanofsky · 2018-03-22T14:26:41Z

src/util.cpp

+int ScheduleBatchPriority(void) {
+#if HAVE_SCHED_H && defined(SCHED_BATCH)
+    sched_param param{.sched_priority = 0};
+    return sched_setscheduler(0, SCHED_BATCH, &param);


Would calling pthread_setschedparam instead of sched_setscheduler be a little more portable? It seems like you might not need the configure check using this.

eklitzke · 2018-03-26T22:58:53Z

Updated to use pthread_setschedparam(). It seems that <sched.h> is standard on Posix systems but SCHED_BATCH is not, so I updated the #ifdef logic a little to reflect that. I also put the error logging in SchedBatchPriority() itself as this return 1 on Windows but errno won't be set. Could also do it the other way (return 0 on non-Linux hosts and then move the log statement to the call site).

While reading another PR I saw a mention of bitcoin#6358. The use case for SCHED_BATCH is to hint to the kernel that the thread is running a non-interactive workload that consumes a lot of CPU time. This is helpful on desktop machines where the loadblk thread can interfere with interactive applications. More details can be found in the sched(7) man page.

laanwj · 2018-04-07T17:24:26Z

utACK d54874d

d54874d Set SCHED_BATCH priority on the loadblk thread. (Evan Klitzke) Pull request description: Today I came across #10271, and while reading the discussion #6358 was linked to. Linux systems have a `SCHED_BATCH` scheduler priority that is useful for threads like loadblk. You can find the full details at [sched(7)](http://man7.org/linux/man-pages/man7/sched.7.html), but I'll quote the relevant part of the man page below: > ...this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wakeup behavior, so that this thread is mildly disfavored in scheduling decisions. > > This policy is useful for workloads that are noninteractive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload's tasks). I think this change is useful independently of #10271 and irrespective of whether that change is merged. Under normal operation the loadblk thread will just import `mempool.dat`. However, if Bitcoin is started with `-reindex` or `-reindex-chainstate` this thread will use a great deal of CPU while it rebuilds the chainstate database (and the block database in the case of `-reindex`). By setting `SCHED_BATCH` this thread is less likely to interfere with interactive tasks (e.g. the user's web browser, text editor, etc.). I'm leaving the nice value unchanged (which also affects scheduling decisions) because I think that's better set by the user. Likewise I'm not using [ioprio_set(2)](http://man7.org/linux/man-pages/man2/ioprio_set.2.html) because it can cause the thread to become completely I/O starved (and knowledgeable users can use `ionice(1)` anyway). Tree-SHA512: ea8f7d3921ed5708948809da771345cdc33efd7ba3323e9dfec07a25bc21e8612e2676f9c178e2710c7bc437e8c9cafc5e0463613688fea5699b6e8e2fec6cff

Summary: d54874d Set SCHED_BATCH priority on the loadblk thread. (Evan Klitzke) Pull request description: Today I came across #10271, and while reading the discussion #6358 was linked to. Linux systems have a `SCHED_BATCH` scheduler priority that is useful for threads like loadblk. You can find the full details at [sched(7)](http://man7.org/linux/man-pages/man7/sched.7.html), but I'll quote the relevant part of the man page below: > ...this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wakeup behavior, so that this thread is mildly disfavored in scheduling decisions. > > This policy is useful for workloads that are noninteractive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload's tasks). I think this change is useful independently of #10271 and irrespective of whether that change is merged. Under normal operation the loadblk thread will just import `mempool.dat`. However, if Bitcoin is started with `-reindex` or `-reindex-chainstate` this thread will use a great deal of CPU while it rebuilds the chainstate database (and the block database in the case of `-reindex`). By setting `SCHED_BATCH` this thread is less likely to interfere with interactive tasks (e.g. the user's web browser, text editor, etc.). I'm leaving the nice value unchanged (which also affects scheduling decisions) because I think that's better set by the user. Likewise I'm not using [ioprio_set(2)](http://man7.org/linux/man-pages/man2/ioprio_set.2.html) because it can cause the thread to become completely I/O starved (and knowledgeable users can use `ionice(1)` anyway). Tree-SHA512: ea8f7d3921ed5708948809da771345cdc33efd7ba3323e9dfec07a25bc21e8612e2676f9c178e2710c7bc437e8c9cafc5e0463613688fea5699b6e8e2fec6cff Backport of Core PR12618 bitcoin/bitcoin#12618 Test Plan: make check test_runner.py Reviewers: deadalnix, Fabien, jasonbcox, O1 Bitcoin ABC, #bitcoin_abc Reviewed By: deadalnix, O1 Bitcoin ABC, #bitcoin_abc Differential Revision: https://reviews.bitcoinabc.org/D3953

Summary: d54874d Set SCHED_BATCH priority on the loadblk thread. (Evan Klitzke) Pull request description: Today I came across #10271, and while reading the discussion #6358 was linked to. Linux systems have a `SCHED_BATCH` scheduler priority that is useful for threads like loadblk. You can find the full details at [sched(7)](http://man7.org/linux/man-pages/man7/sched.7.html), but I'll quote the relevant part of the man page below: > ...this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wakeup behavior, so that this thread is mildly disfavored in scheduling decisions. > > This policy is useful for workloads that are noninteractive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload's tasks). I think this change is useful independently of #10271 and irrespective of whether that change is merged. Under normal operation the loadblk thread will just import `mempool.dat`. However, if Bitcoin is started with `-reindex` or `-reindex-chainstate` this thread will use a great deal of CPU while it rebuilds the chainstate database (and the block database in the case of `-reindex`). By setting `SCHED_BATCH` this thread is less likely to interfere with interactive tasks (e.g. the user's web browser, text editor, etc.). I'm leaving the nice value unchanged (which also affects scheduling decisions) because I think that's better set by the user. Likewise I'm not using [ioprio_set(2)](http://man7.org/linux/man-pages/man2/ioprio_set.2.html) because it can cause the thread to become completely I/O starved (and knowledgeable users can use `ionice(1)` anyway). Tree-SHA512: ea8f7d3921ed5708948809da771345cdc33efd7ba3323e9dfec07a25bc21e8612e2676f9c178e2710c7bc437e8c9cafc5e0463613688fea5699b6e8e2fec6cff Backport of Core PR12618 bitcoin/bitcoin#12618 Test Plan: make check test_runner.py Reviewers: deadalnix, Fabien, jasonbcox, O1 Bitcoin ABC, #bitcoin_abc Reviewed By: deadalnix, O1 Bitcoin ABC, #bitcoin_abc Differential Revision: https://reviews.bitcoinabc.org/D3953

d54874d Set SCHED_BATCH priority on the loadblk thread. (Evan Klitzke) Pull request description: Today I came across bitcoin#10271, and while reading the discussion dashpay#6358 was linked to. Linux systems have a `SCHED_BATCH` scheduler priority that is useful for threads like loadblk. You can find the full details at [sched(7)](http://man7.org/linux/man-pages/man7/sched.7.html), but I'll quote the relevant part of the man page below: > ...this policy will cause the scheduler to always assume that the thread is CPU-intensive. Consequently, the scheduler will apply a small scheduling penalty with respect to wakeup behavior, so that this thread is mildly disfavored in scheduling decisions. > > This policy is useful for workloads that are noninteractive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload's tasks). I think this change is useful independently of bitcoin#10271 and irrespective of whether that change is merged. Under normal operation the loadblk thread will just import `mempool.dat`. However, if Bitcoin is started with `-reindex` or `-reindex-chainstate` this thread will use a great deal of CPU while it rebuilds the chainstate database (and the block database in the case of `-reindex`). By setting `SCHED_BATCH` this thread is less likely to interfere with interactive tasks (e.g. the user's web browser, text editor, etc.). I'm leaving the nice value unchanged (which also affects scheduling decisions) because I think that's better set by the user. Likewise I'm not using [ioprio_set(2)](http://man7.org/linux/man-pages/man2/ioprio_set.2.html) because it can cause the thread to become completely I/O starved (and knowledgeable users can use `ionice(1)` anyway). Tree-SHA512: ea8f7d3921ed5708948809da771345cdc33efd7ba3323e9dfec07a25bc21e8612e2676f9c178e2710c7bc437e8c9cafc5e0463613688fea5699b6e8e2fec6cff

fanquake added the Resource usage label Mar 6, 2018

laanwj reviewed Mar 6, 2018

View reviewed changes

eklitzke mentioned this pull request Mar 15, 2018

Proposal: Refactoring platform-specific code in util.h/util.cpp #12697

Closed

maflcko added this to the 0.17.0 milestone Mar 18, 2018

eklitzke force-pushed the sched_batch branch from aaa2228 to a2ed276 Compare March 19, 2018 06:56

eklitzke mentioned this pull request Mar 20, 2018

Drop IO priority to idle while reading blocks for peer requests and startup verification #9245

Closed

ryanofsky reviewed Mar 22, 2018

View reviewed changes

eklitzke force-pushed the sched_batch branch 3 times, most recently from 0d5541e to 8fe7e95 Compare March 26, 2018 22:55

eklitzke force-pushed the sched_batch branch from 8fe7e95 to d54874d Compare March 26, 2018 22:59

laanwj merged commit d54874d into bitcoin:master Apr 7, 2018

maflcko mentioned this pull request Apr 8, 2018

Segmentation fault in util: ScheduleBatchPriority #12915

Closed

dagurval mentioned this pull request Jun 11, 2018

Default to virtual cores and set appropriate thread scheduler bitcoinxt/bitcoinxt#446

Merged

sickpig mentioned this pull request Jun 19, 2018

[PORT] Default to virtual cores and set appropriate thread scheduler BitcoinUnlimited/BitcoinUnlimited#1151

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Set SCHED_BATCH priority on the loadblk thread. #12618

Set SCHED_BATCH priority on the loadblk thread. #12618

Uh oh!

eklitzke commented Mar 6, 2018

Uh oh!

laanwj Mar 6, 2018

Uh oh!

eklitzke Mar 6, 2018

Uh oh!

laanwj commented Mar 6, 2018

Uh oh!

promag commented Mar 6, 2018

Uh oh!

eklitzke commented Mar 6, 2018

Uh oh!

JeremyRubin commented Mar 6, 2018

Uh oh!

eklitzke commented Mar 9, 2018

Uh oh!

maflcko commented Mar 18, 2018

Uh oh!

eklitzke commented Mar 19, 2018

Uh oh!

ryanofsky left a comment

Uh oh!

ryanofsky Mar 22, 2018

Uh oh!

ryanofsky Mar 22, 2018

Uh oh!

eklitzke commented Mar 26, 2018

Uh oh!

laanwj commented Apr 7, 2018

Uh oh!

Uh oh!

Set SCHED_BATCH priority on the loadblk thread. #12618

Set SCHED_BATCH priority on the loadblk thread. #12618

Uh oh!

Conversation

eklitzke commented Mar 6, 2018

Uh oh!

laanwj Mar 6, 2018

Choose a reason for hiding this comment

Uh oh!

eklitzke Mar 6, 2018

Choose a reason for hiding this comment

Uh oh!

laanwj commented Mar 6, 2018

Uh oh!

promag commented Mar 6, 2018

Uh oh!

eklitzke commented Mar 6, 2018

Uh oh!

JeremyRubin commented Mar 6, 2018

Uh oh!

eklitzke commented Mar 9, 2018

Uh oh!

maflcko commented Mar 18, 2018

Uh oh!

eklitzke commented Mar 19, 2018

Uh oh!

ryanofsky left a comment

Choose a reason for hiding this comment

Uh oh!

ryanofsky Mar 22, 2018

Choose a reason for hiding this comment

Uh oh!

ryanofsky Mar 22, 2018

Choose a reason for hiding this comment

Uh oh!

eklitzke commented Mar 26, 2018

Uh oh!

laanwj commented Apr 7, 2018

Uh oh!

Uh oh!