Faster archiving for non-days periods by only storing one datatable and blob row at a time in memory #20512
+485
−220
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers |
…ow in memory at a time (#20332) * proof of concept one blob table row at a time method of aggregating week, month, year, range data * sort blobs by subtable ID when chunk is being read * simplify code w/ generators * make sure single blob query is ordered by name correctly * REGEXP_SUBSTR() is only available in mysql 8 :/ * fix a couple test failures * by default when aggregating tables in ArchiveProcessor sort by visits if the table contains visits * try fixing random test failures * debug ci only error * undo debugging change * fixing some system tests * refactor ArchiveSelector code for more code reuse * add some code documentation * remove DataCollection::forEachBlobExpanded() since it is no longer used + a couple other small changes * try debugging ci only random failure * remove previous debugging code * more debugging * more ci debugging * trigger build again and try to get more information for random failure * fix convoluted sql replacement for REGEXP_SUBSTRING * fix idsubtable extraction, need to check if extracted value is an empty string and order it before everything else if so * add log in case blob table order is incorrect * add tests for subtable extraction sql * remove unused import --------- Co-authored-by: Stefan Giehl <stefan@matomo.org>
sgiehl
approved these changes
Apr 12, 2023
diosmosis
added a commit
to matomo-org/plugin-GoogleAnalyticsImporter
that referenced
this pull request
Apr 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
refs #20332. This was merged in
5.x-dev. It would be a huge help for us on the Cloud to have this merged in4.x-devas well until 5.0 is released. From what I can see there are no breaking changes? In that case it be great to also apply this to 4.x.Review