Skip to content

DB: Split log_* Tracker tables into hot/cold for faster tracking API requests & reports archiving  #14119

@mattab

Description

@mattab

In order to improve overall performance of Matomo (both at Tracker and Reports processing) an idea we have is to split log_* Tracker tables into two sets of tables: hot data VS cold data.

eg hot data would be data for the last 48 hours or so. Cold data would be any data older than 48 hours. As a result the "hot data" table could be 100 times smaller than the cold data (depending on data retention settings see https://matomo.org/docs/managing-your-databases-size/ )

Having tables split in this way means that:

  • Tracking requests is faster (inserting into smaller tables takes less time)
  • Archiving reports is faster (querying the same datasets but over smaller log_ tables will be faster)

There will be a few changes needed eg. need a process to move data from hot to cold, need to update SQL query generators to query the right tables as applicable (and possibly getting data from the two tables eg. Visitor log requests over multiple days), etc.

Metadata

Metadata

Assignees

No one assigned

    Labels

    MajorIndicates the severity or impact or benefit of an issue is much higher than normal but not critical.c: PerformanceFor when we could improve the performance / speed of Matomo.wontfixIf you can reproduce this issue, please reopen the issue or create a new one describing it.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions