Start one background worker per database #545
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
56429f3 to
0a0f3c4
Compare
2b13b4c to
9afec02
Compare
9e1ac92 to
f2834ac
Compare
eaf7065 to
d41ee16
Compare
Y--
added a commit
that referenced
this pull request
Mar 10, 2025
Up until now, we were starting the background worker as part of the extension. The problem with this approach is that it imposes a unique worker, and thus cannot updated multiple database. In preparation to have one BGW per database, this PR triggers the start of a background worker when `IsExtensionRegistered` is called and the cache is populated. We've prevented potential race conditions that would cause two bgw to start by checking a lock on a temporary file. Note: a [following PR](#545) will actually start one BGW per database --------- Co-authored-by: Jelte Fennema-Nio <github-tech@jeltef.nl>
JelteF
reviewed
Mar 11, 2025
6b09a92 to
a4adf47
Compare
JelteF
requested changes
Mar 13, 2025
JelteF
reviewed
Apr 14, 2025
Y--
commented
Apr 16, 2025
Y--
commented
Apr 16, 2025
| @@ -417,11 +451,9 @@ TriggerActivity(void) { | |||
| } | |||
|
|
|||
| SpinLockAcquire(&BgwShmemStruct->lock); | |||
| BgwShmemStruct->activity_count++; | |||
| /* Force wake up the background worker */ | |||
| if (BgwShmemStruct->bgw_latch) { | |||
There was a problem hiding this comment.
This if meant that somehow we were notifying changes before the bgw was available.
This should not happen anymore.
JelteF
reviewed
Apr 16, 2025
Y--
added a commit
that referenced
this pull request
Apr 17, 2025
In order to correctly test #545 we will need to have multiple test users. This PR paves the way to generate as many MotherDuck test user as we want. I am considering removing the support for `MOTHERDUCK_TEST_TOKEN`, but I have kept it for now. The main change is now we can run the test suite with a token that can generate test users and it will generates one and use it through the test suite.
JelteF
approved these changes
Apr 22, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Following #544, remove the
duckdb.motherduck_postgres_databaseGUC parameter and start a background worker for each PG database (*)For another database to synchronize with the BGW to work:
CREATE EXTENSION pg_duckdb;duckdbshould be able to write inpublicschema:GRANT ALL ON SCHEMA public to duckdb;