Edit: Tracking expected mitigations in:

• Lower minumum batch size to 1 for background updates #11422
• Improve performance of remove_{hidden,deleted}_devices_from_device_inbox #11421

See discussion below for context

Description

After upgrading to synapse-1.47, via https://github.com/spantaleev/matrix-docker-ansible-deploy, my server experienced a huge increase in both CPU and IO load.

Upon examining the synapse prometheus/grafana stats, I found the following entires:

A large increase in master_0_background_updates in DB transactions

And in particular, master-0_remove_hidden_device_from_inbox

It may be related, that the HTTP pusher distribution also changed oddly:

Steps to reproduce

• upgraded from 1.46 to 1.47

Version information

• Version: 1.47

• Install method: https://github.com/spantaleev/matrix-docker-ansible-deploy

• Platform: Ubuntu 20.04.3