User Story

As an: advanced user / system admin / IT support person
I want to: be able to quickly check the status of archiving for a Matomo instance
So that: I can understand the archiving load and diagnose any performance issues

Summary

It's currently difficult to analyze the state of archiving for a Matomo instance without running custom SQL queries, checking logs or config settings, all of which require advanced knowledge of the archiving process.

Console commands could be added which will automatically gather all relevant diagnostic information for archiving on the current instance:

./console diagnostics:archiving-status [--with-stats] - Show all archiving diagnostics combined into one output, no instance statistics unless explicitly specified with the optional flag.
./console diagnostics:archiving-queue [--json] - Show only the current archive queue as a table, optionally as JSON
./console diagnostics:archiving-metrics [--json] - Show only archiving metrics, optionally as JSON (for monitoring)
./console diagnostics:archiving-queue-max-wait - Return the current queue max wait mins as an integer (for monitoring)
./console diagnostics:archiving-queue-size - Return the current queue length as an integer (for monitoring)
./console diagnostics:archiving-last-archive-created - Return number of minutes since the last archive create time (intended for monitoring)

These commands could be used in the following scenarios:

• Direct troubleshooting.
• Remote debugging where on-premise users can easily retrieve diagnostic information to send to support.
• Automated monitoring where a command is called by a scheduled job and the output sent to a monitoring tool.

Diagnostics

The current archive queue

A list of data from the archive invalidations table showing a descriptive name, site, date, period, time queued, time waiting and status, sorted by oldest.

Archiving metrics

A snapshot of the current queue metrics:
Longest waiting mins : 40
Queue size: 16
Last archived created date: 2023-12-25 11:59:59 (SELECT MAX(ts_archived) FROM archive_blob_year_month;)
Last archived created mins ago: 200

Instance statistics

General information about the instance that can affect archiving performance, excluded by default from the archiving-status command for performance / privacy reasons:

• Site count (SELECT COUNT(*) FROM site;)
• Segment count (SELECT COUNT(*) FROM segment;
• Database engine and version
• Log data size
  • SELECT COUNT(*) FROM log_visit;
  • SELECT COUNT(*) FROM log_actions;
  • SELECT COUNT(*) FROM log_conversions;

Archiving config

Current values for all archive related configuration settings:

enable_segment_first_table_join_prefix
enable_first_table_join_prefix
enable_processing_unique_visitors_day
enable_processing_unique_visitors_week
enable_processing_unique_visitors_month
enable_processing_unique_visitors_year
enable_processing_unique_visitors_range
enable_processing_unique_visitors_multiple_sites
process_new_segments_from
time_before_today_archive_considered_outdated
time_before_week_archive_considered_outdated
time_before_month_archive_considered_outdated
time_before_year_archive_considered_outdated
time_before_range_archive_considered_outdated
enable_browser_archiving_triggering
archiving_range_force_on_browser_request
archiving_custom_ranges[]
archiving_query_max_execution_time
disable_archiving_segment_for_plugins
disable_archive_actions_goals
datatable_archiving_maximum_*
archiving_ranking_query_row_limit