Matomo provides a fast way to insert many archives using load data infile see https://matomo.org/faq/troubleshooting/faq_194/

This works when calling insertBlobRecord https://github.com/matomo-org/matomo/blob/3.12.0-b2/core/DataAccess/ArchiveWriter.php#L95

However, it works only when inserting multiple records at once. Mostly, we only insert one data table / archive at once and therefore it never makes use of this performance optimisation.

The idea be to not directly insert archives there, but only group insert them once there are eg 10 or 20 or 50 archives. This shouldn't be a memory issue but we may need to check that we don't hold too many data in memory.

Login be like this:

• Call insertBlobRecord($records)
• Add it to an archive cache like array_merge($this->archivesToInsert, $records)
• Are there more than say 30 records in $this->archivesToInsert ?
  • Yes => Actually insert them
  • No => Do not insert them
• After archiving, when calling finalizeArchive() we would insert all remaining archives that haven't been inserted yet.

We need to figure out re memory what is a good size of number of dataTables to store in a cache... I reckon 10 should be fine for sure, possibly even 50 or 100.