This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Synapse makes postgres to eat all ram until OOM #9182
Description
Description
Synapse makes related postgres processes to slowly eat all ram (16g) and swap until OOM hits.
Steps to reproduce
For this I do'nt have anything to really tell, will provide tried things later on.
Version information
- hacklab.fi
- Synapse 1.25.0
- Postgres 13 (same issue with 12)
- Debuan buster
- matrix.org provided Synapse repo
When postgres and synapse is started, all synapse related postgres threads has RES quite exactly 250mb, after 12h or so their RES is approaching 1200mb each, median 1100. Obviously I have tried all the normal postgres tunables but nothing really makes perceivable difference on this behaviour. Restarting either synapse or postgres process will make memory go back to normal RES, just to climb away to OOM again
I have tried to dig what could possibly cause this behaviour, but at this point I an out of ideas. On comparable but maybe doubly bigger HS kapsi.fi that does have redis workers and whatnot enabled, does not experience this behaviour, postgres process is well happy with memory pretty much it has at startup.
Here is postgres=# select datname, usename, pid, backend_start, xact_start, query_start, state_change, wait_event_type, wait_event, state, query from pg_stat_activity;
datname | usename | pid | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | state | query
-------------+--------------+--------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+---------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| postgres | 172769 | 2021-01-20 10:02:07.119053+02 | | | | Activity | LogicalLauncherMain | |
synapse | synapse_user | 172776 | 2021-01-20 10:02:08.283794+02 | | 2021-01-20 19:31:02.698945+02 | 2021-01-20 19:31:02.698961+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172774 | 2021-01-20 10:02:08.277446+02 | | 2021-01-20 19:31:02.693968+02 | 2021-01-20 19:31:02.693981+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172775 | 2021-01-20 10:02:08.277738+02 | | 2021-01-20 19:31:02.703326+02 | 2021-01-20 19:31:02.70335+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172778 | 2021-01-20 10:02:08.28457+02 | | 2021-01-20 19:31:02.695861+02 | 2021-01-20 19:31:02.695879+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172777 | 2021-01-20 10:02:08.284047+02 | | 2021-01-20 19:31:02.697951+02 | 2021-01-20 19:31:02.697974+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172779 | 2021-01-20 10:02:08.303174+02 | | 2021-01-20 19:31:02.691738+02 | 2021-01-20 19:31:02.691757+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172780 | 2021-01-20 10:02:08.313032+02 | | 2021-01-20 19:31:02.692357+02 | 2021-01-20 19:31:02.692368+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172781 | 2021-01-20 10:02:08.313392+02 | | 2021-01-20 19:31:02.691576+02 | 2021-01-20 19:31:02.691586+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172782 | 2021-01-20 10:02:08.320273+02 | | 2021-01-20 19:31:02.690884+02 | 2021-01-20 19:31:02.690911+02 | Client | ClientRead | idle | COMMIT
synapse | synapse_user | 172783 | 2021-01-20 10:02:08.321661+02 | | 2021-01-20 19:31:02.693378+02 | 2021-01-20 19:31:02.693389+02 | Client | ClientRead | idle | COMMIT
telegrambot | telegrambot | 182339 | 2021-01-20 19:05:41.200507+02 | | 2021-01-20 19:30:59.350267+02 | 2021-01-20 19:30:59.350449+02 | Client | ClientRead | idle | SELECT pg_advisory_unlock_all(); +
| | | | | | | | | | CLOSE ALL; +
| | | | | | | | | | +
| | | | | | | | | | DO $$ +
| | | | | | | | | | BEGIN +
| | | | | | | | | | PERFORM * FROM pg_listening_channels() LIMIT 1; +
| | | | | | | | | | IF FOUND THEN +
| | | | | | | | | | UNLISTEN *; +
| | | | | | | | | | END IF; +
| | | | | | | | | | END; +
| | | | | | | | | | $$; +
| | | | | | | | | | +
| | | | | | | | | | RESET ALL;
postgres | postgres | 182778 | 2021-01-20 19:30:59.687796+02 | 2021-01-20 19:31:02.683108+02 | 2021-01-20 19:31:02.683108+02 | 2021-01-20 19:31:02.683109+02 | | | active | select datname, usename, pid, backend_start, xact_start, query_start, state_change, wait_event_type, wait_event, state, query from pg_stat_activity;
facebookbot | facebookbot | 172786 | 2021-01-20 10:02:08.41789+02 | | 2021-01-20 19:30:32.971077+02 | 2021-01-20 19:30:32.971258+02 | Client | ClientRead | idle | SELECT pg_advisory_unlock_all(); +
| | | | | | | | | | CLOSE ALL; +
| | | | | | | | | | +
| | | | | | | | | | DO $$ +
| | | | | | | | | | BEGIN +
| | | | | | | | | | PERFORM * FROM pg_listening_channels() LIMIT 1; +
| | | | | | | | | | IF FOUND THEN +
| | | | | | | | | | UNLISTEN *; +
| | | | | | | | | | END IF; +
| | | | | | | | | | END; +
| | | | | | | | | | $$; +
| | | | | | | | | | +
| | | | | | | | | | RESET ALL;
telegrambot | telegrambot | 172794 | 2021-01-20 10:02:09.385851+02 | | 2021-01-20 19:28:32.072588+02 | 2021-01-20 19:28:32.074156+02 | Client | ClientRead | idle | COMMIT
grafana | grafana | 181467 | 2021-01-20 18:18:29.98819+02 | | 2021-01-20 19:31:00.981616+02 | 2021-01-20 19:31:00.981806+02 | Client | ClientRead | idle | select * from alert
telegrambot | telegrambot | 172802 | 2021-01-20 10:02:20.980309+02 | | 2021-01-20 19:28:06.589256+02 | 2021-01-20 19:28:06.589273+02 | Client | ClientRead | idle | COMMIT
telegrambot | telegrambot | 172803 | 2021-01-20 10:02:20.997652+02 | | 2021-01-20 19:28:32.168638+02 | 2021-01-20 19:28:32.170706+02 | Client | ClientRead | idle | COMMIT
telegrambot | telegrambot | 172804 | 2021-01-20 10:02:21.01352+02 | | 2021-01-20 19:28:32.171649+02 | 2021-01-20 19:28:32.171689+02 | Client | ClientRead | idle | COMMIT
telegrambot | telegrambot | 172805 | 2021-01-20 10:02:21.023916+02 | | 2021-01-20 19:28:32.076235+02 | 2021-01-20 19:28:32.076275+02 | Client | ClientRead | idle | ROLLBACK
signalbot | signalbot | 172813 | 2021-01-20 10:02:32.943974+02 | | 2021-01-20 19:30:45.81808+02 | 2021-01-20 19:30:45.81825+02 | Client | ClientRead | idle | SELECT pg_advisory_unlock_all(); +
| | | | | | | | | | CLOSE ALL; +
| | | | | | | | | | +
| | | | | | | | | | DO $$ +
| | | | | | | | | | BEGIN +
| | | | | | | | | | PERFORM * FROM pg_listening_channels() LIMIT 1; +
| | | | | | | | | | IF FOUND THEN +
| | | | | | | | | | UNLISTEN *; +
| | | | | | | | | | END IF; +
| | | | | | | | | | END; +
| | | | | | | | | | $$; +
| | | | | | | | | | +
| | | | | | | | | | RESET ALL;
facebookbot | facebookbot | 172866 | 2021-01-20 10:05:53.116227+02 | | 2021-01-20 19:08:07.541813+02 | 2021-01-20 19:08:07.541822+02 | Client | ClientRead | idle | ROLLBACK
grafana | grafana | 181810 | 2021-01-20 18:38:29.988601+02 | | 2021-01-20 19:30:50.981635+02 | 2021-01-20 19:30:50.981968+02 | Client | ClientRead | idle | select * from alert
whatsappbot | whatsappbot | 182449 | 2021-01-20 19:13:13.567375+02 | | 2021-01-20 19:30:52.996283+02 | 2021-01-20 19:30:52.997808+02 | Client | ClientRead | idle | UPDATE puppet SET displayname=$1, name_quality=$2, avatar=$3, avatar_url=$4, custom_mxid=$5, access_token=$6, next_batch=$7, enable_presence=$8, enable_receipts=$9 WHERE jid=$10
whatsappbot | whatsappbot | 182441 | 2021-01-20 19:13:13.551931+02 | | 2021-01-20 19:30:57.959069+02 | 2021-01-20 19:30:57.960742+02 | Client | ClientRead | idle | UPDATE puppet SET displayname=$1, name_quality=$2, avatar=$3, avatar_url=$4, custom_mxid=$5, access_token=$6, next_batch=$7, enable_presence=$8, enable_receipts=$9 WHERE jid=$10
| | 172766 | 2021-01-20 10:02:07.118342+02 | | | | Activity | BgWriterMain | |
| | 172765 | 2021-01-20 10:02:07.118096+02 | | | | Activity | CheckpointerMain | |
| | 172767 | 2021-01-20 10:02:07.118604+02 | | | | Activity | WalWriterMain | |
(28 rows)
I've also tried to run (gdb) p MemoryContextStats(TopMemoryContext) onto one such process, here are results from startup and then when ram almost eaten:
TopMemoryContext: 154592 total in 8 blocks; 47496 free (98 chunks); 107096 used
pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1448 free (0 chunks); 6744 used
HandleParallelMessages: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
RI compare cache: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used
RI query cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
RI constraint cache: 40888 total in 2 blocks; 2616 free (0 chunks); 38272 used
Sequence values: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
Btree proof lookup cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
CFuncHash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
Tsearch dictionary cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
Tsearch parser cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
Tsearch configuration cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
TableSpace cache: 8192 total in 1 blocks; 2088 free (0 chunks); 6104 used
Type information cache: 24616 total in 2 blocks; 2616 free (0 chunks); 22000 used
Operator lookup cache: 24576 total in 2 blocks; 10752 free (3 chunks); 13824 used
RowDescriptionContext: 8192 total in 1 blocks; 6888 free (0 chunks); 1304 used
MessageContext: 8192 total in 1 blocks; 6888 free (1 chunks); 1304 used
Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
smgr relation table: 131072 total in 5 blocks; 73928 free (19 chunks); 57144 used
TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
TopPortalContext: 8192 total in 1 blocks; 7928 free (1 chunks); 264 used
Relcache by OID: 32768 total in 3 blocks; 10488 free (6 chunks); 22280 used
CacheMemoryContext: 4689520 total in 32 blocks; 1746360 free (0 chunks); 2943160 used
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: room_retention_max_lifetime_idx
index info: 2048 total in 2 blocks; 608 free (1 chunks); 1440 used: room_retention_pkey
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: pg_toast_17486_index
index info: 3072 total in 2 blocks; 1080 free (1 chunks); 1992 used: room_tag_uniqueness
CachedPlan: 8192 total in 4 blocks; 3464 free (0 chunks); 4728 used: SELECT 1 FROM ONLY "public"."access_tokens" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF ...
CachedPlan: 8192 total in 4 blocks; 2864 free (0 chunks); 5328 used: SELECT 1 FROM ONLY "public"."events" x WHERE "event_id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x
index info: 2048 total in 2 blocks; 872 free (0 chunks); 1176 used: event_push_summary_stream_ordering_lock_key
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: ex_outlier_stream_pkey
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: group_attestations_renewals_v_idx
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: group_attestations_renewals_u_idx
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: group_attestations_renewals_g_idx
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: monthly_active_users_users
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: monthly_active_users_time_stamp
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_signature_stream_idx
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: ui_auth_sessions_session_id_key
index info: 3072 total in 2 blocks; 664 free (1 chunks); 2408 used: e2e_one_time_keys_json_uniqueness
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: e2e_room_keys_versions_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: access_tokens_device_id
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: access_tokens_token_key
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: access_tokens_pkey
CachedPlanSource: 4096 total in 3 blocks; 1416 free (0 chunks); 2680 used: SELECT 1 FROM ONLY "public"."access_tokens" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF ...
CachedPlanQuery: 4096 total in 3 blocks; 1192 free (1 chunks); 2904 used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlanSource: 4096 total in 3 blocks; 1416 free (0 chunks); 2680 used: SELECT 1 FROM ONLY "public"."events" x WHERE "event_id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x
CachedPlanQuery: 4096 total in 3 blocks; 592 free (0 chunks); 3504 used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: pg_toast_17607_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: state_events_event_id_key
index info: 2048 total in 2 blocks; 840 free (0 chunks); 1208 used: evauth_edges_id
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: local_current_membership_room_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: local_current_membership_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: pg_toast_17742_index
TS dictionary: 1024 total in 1 blocks; 688 free (0 chunks); 336 used: simple
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: room_stats_state_room
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: device_lists_remote_cache_unique_id
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: user_daily_visits_uts_idx
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: user_daily_visits_ts_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: state_groups_room_id_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: state_groups_pkey
index info: 3072 total in 2 blocks; 968 free (1 chunks); 2104 used: users_who_share_private_rooms_u_idx
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: users_who_share_private_rooms_r_idx
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: users_who_share_private_rooms_o_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: users_in_public_rooms_u_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: users_in_public_rooms_r_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_directory_search_user_idx
index info: 4096 total in 3 blocks; 2256 free (2 chunks); 1840 used: user_directory_search_fts_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_directory_user_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_directory_room_idx
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: erased_users_user
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: group_attestations_remote_v_idx
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: group_attestations_remote_u_idx
index info: 2048 total in 2 blocks; 608 free (1 chunks); 1440 used: group_attestations_remote_g_idx
index info: 2048 total in 2 blocks; 656 free (1 chunks); 1392 used: group_roles_group_id_role_id_key
index info: 3072 total in 2 blocks; 1048 free (1 chunks); 2024 used: group_summary_roles_group_id_role_id_role_order_key
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: group_summary_users_g_idx
index info: 2048 total in 2 blocks; 656 free (1 chunks); 1392 used: group_room_categories_group_id_category_id_key
index info: 3072 total in 2 blocks; 1048 free (1 chunks); 2024 used: group_summary_room_categories_group_id_category_id_cat_orde_key
index info: 3072 total in 2 blocks; 1080 free (1 chunks); 1992 used: group_summary_rooms_g_idx
index info: 3072 total in 2 blocks; 1064 free (1 chunks); 2008 used: group_summary_rooms_group_id_category_id_room_id_room_order_key
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: group_rooms_r_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: group_rooms_g_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_stats_historical_end_ts
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: user_stats_historical_pkey
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_stats_current_pkey
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: current_state_delta_stream_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: device_federation_outbox_id
index info: 2048 total in 2 blocks; 496 free (1 chunks); 1552 used: device_federation_outbox_destination_id
index info: 2048 total in 2 blocks; 496 free (1 chunks); 1552 used: device_lists_outbound_pokes_user
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: device_lists_outbound_pokes_stream
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: device_lists_outbound_pokes_id
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: cache_invalidation_stream_by_instance_id
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: e2e_cross_signing_keys_stream_idx
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: e2e_cross_signing_keys_idx
index info: 2048 total in 2 blocks; 920 free (0 chunks); 1128 used: groups_idx
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: event_relations_relates
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: event_relations_id
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: local_group_membership_u_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: local_group_membership_g_idx
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: server_keys_json_uniqueness
index info: 2048 total in 2 blocks; 872 free (0 chunks); 1176 used: private_user_data_max_stream_id_lock_key
index info: 3072 total in 2 blocks; 696 free (1 chunks); 2376 used: event_txn_id_txn_id
index info: 2048 total in 2 blocks; 840 free (0 chunks); 1208 used: event_txn_id_ts
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: event_txn_id_event_id
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: stream_positions_idx
index info: 3072 total in 2 blocks; 696 free (1 chunks); 2376 used: e2e_room_keys_with_version_idx
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: device_inbox_user_stream_id
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: device_inbox_stream_id_user_id
index info: 2048 total in 2 blocks; 608 free (1 chunks); 1440 used: room_tag_revisions_uniqueness
index info: 4096 total in 3 blocks; 2184 free (2 chunks); 1912 used: room_memberships_user_room_forgotten
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: room_memberships_user_id
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: room_memberships_room_id
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: room_memberships_event_id_key
index info: 3544 total in 3 blocks; 432 free (0 chunks); 3112 used: remote_media_repository_thumbn_media_origin_id_width_height_met
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: public_room_index
index info: 2048 total in 2 blocks; 840 free (0 chunks); 1208 used: rooms_pkey
index info: 2048 total in 2 blocks; 840 free (0 chunks); 1208 used: pushers2_pkey
index info: 3072 total in 2 blocks; 808 free (1 chunks); 2264 used: pushers2_app_id_pushkey_user_name_key
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: application_services_txns_id
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: application_services_txns_as_id_txn_id_key
index info: 2048 total in 2 blocks; 872 free (0 chunks); 1176 used: user_directory_stream_pos_lock_key
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: room_account_data_stream_id
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: room_account_data_uniqueness
187 more child contexts containing 402664 total in 360 blocks; 126120 free (131 chunks); 276544 used
WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used
PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
MdSmgr: 16384 total in 2 blocks; 3944 free (3 chunks); 12440 used
LOCALLOCK hash: 32768 total in 3 blocks; 16824 free (8 chunks); 15944 used
Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
ErrorContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
Grand total: 6137360 bytes in 653 blocks; 2226664 free (318 chunks); 3910696 used
TopMemoryContext: 154592 total in 8 blocks; 47496 free (98 chunks); 107096 used
pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1448 free (0 chunks); 6744 used
HandleParallelMessages: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
RI compare cache: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used
RI query cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
RI constraint cache: 40888 total in 2 blocks; 2616 free (0 chunks); 38272 used
Sequence values: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
Btree proof lookup cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
CFuncHash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
Tsearch dictionary cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
Tsearch parser cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
Tsearch configuration cache: 8192 total in 1 blocks; 1576 free (0 chunks); 6616 used
TableSpace cache: 8192 total in 1 blocks; 2088 free (0 chunks); 6104 used
Type information cache: 24616 total in 2 blocks; 2616 free (0 chunks); 22000 used
Operator lookup cache: 24576 total in 2 blocks; 10752 free (3 chunks); 13824 used
RowDescriptionContext: 8192 total in 1 blocks; 6888 free (0 chunks); 1304 used
MessageContext: 8192 total in 1 blocks; 6888 free (1 chunks); 1304 used
Operator class cache: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
smgr relation table: 131072 total in 5 blocks; 73928 free (19 chunks); 57144 used
TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
Portal hash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
TopPortalContext: 8192 total in 1 blocks; 7928 free (1 chunks); 264 used
Relcache by OID: 32768 total in 3 blocks; 9448 free (6 chunks); 23320 used
CacheMemoryContext: 4722032 total in 34 blocks; 1659648 free (1 chunks); 3062384 used
index info: 3072 total in 2 blocks; 1048 free (1 chunks); 2024 used: e2e_fallback_keys_json_uniqueness
index info: 2048 total in 2 blocks; 872 free (0 chunks); 1176 used: threepid_validation_token_session_id
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: threepid_validation_token_pkey
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: device_federation_inbox_sender_id
index info: 2048 total in 2 blocks; 920 free (0 chunks); 1128 used: room_aliases_id
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: room_aliases_room_alias_key
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: blocked_rooms_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: pg_toast_17662_index
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: room_retention_max_lifetime_idx
index info: 2048 total in 2 blocks; 608 free (1 chunks); 1440 used: room_retention_pkey
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: pg_toast_17486_index
index info: 3072 total in 2 blocks; 1080 free (1 chunks); 1992 used: room_tag_uniqueness
CachedPlan: 8192 total in 4 blocks; 3464 free (0 chunks); 4728 used: SELECT 1 FROM ONLY "public"."access_tokens" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF ...
CachedPlan: 8192 total in 4 blocks; 2864 free (0 chunks); 5328 used: SELECT 1 FROM ONLY "public"."events" x WHERE "event_id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x
index info: 2048 total in 2 blocks; 872 free (0 chunks); 1176 used: event_push_summary_stream_ordering_lock_key
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: ex_outlier_stream_pkey
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: group_attestations_renewals_v_idx
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: group_attestations_renewals_u_idx
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: group_attestations_renewals_g_idx
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: monthly_active_users_users
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used: monthly_active_users_time_stamp
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_signature_stream_idx
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: ui_auth_sessions_session_id_key
index info: 3072 total in 2 blocks; 664 free (1 chunks); 2408 used: e2e_one_time_keys_json_uniqueness
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: e2e_room_keys_versions_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: access_tokens_device_id
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: access_tokens_token_key
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: access_tokens_pkey
CachedPlanSource: 4096 total in 3 blocks; 1416 free (0 chunks); 2680 used: SELECT 1 FROM ONLY "public"."access_tokens" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF ...
CachedPlanQuery: 4096 total in 3 blocks; 1192 free (1 chunks); 2904 used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
CachedPlanSource: 4096 total in 3 blocks; 1416 free (0 chunks); 2680 used: SELECT 1 FROM ONLY "public"."events" x WHERE "event_id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x
CachedPlanQuery: 4096 total in 3 blocks; 592 free (0 chunks); 3504 used
SPI Plan: 1024 total in 1 blocks; 576 free (0 chunks); 448 used
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: pg_toast_17607_index
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: state_events_event_id_key
index info: 2048 total in 2 blocks; 840 free (0 chunks); 1208 used: evauth_edges_id
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: local_current_membership_room_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: local_current_membership_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: pg_toast_17742_index
TS dictionary: 1024 total in 1 blocks; 688 free (0 chunks); 336 used: simple
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: room_stats_state_room
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: device_lists_remote_cache_unique_id
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: user_daily_visits_uts_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_daily_visits_ts_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: state_groups_room_id_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: state_groups_pkey
index info: 3072 total in 2 blocks; 968 free (1 chunks); 2104 used: users_who_share_private_rooms_u_idx
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: users_who_share_private_rooms_r_idx
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: users_who_share_private_rooms_o_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: users_in_public_rooms_u_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: users_in_public_rooms_r_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_directory_search_user_idx
index info: 4096 total in 3 blocks; 2256 free (2 chunks); 1840 used: user_directory_search_fts_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_directory_user_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_directory_room_idx
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: erased_users_user
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: group_attestations_remote_v_idx
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: group_attestations_remote_u_idx
index info: 2048 total in 2 blocks; 608 free (1 chunks); 1440 used: group_attestations_remote_g_idx
index info: 2048 total in 2 blocks; 656 free (1 chunks); 1392 used: group_roles_group_id_role_id_key
index info: 3072 total in 2 blocks; 1048 free (1 chunks); 2024 used: group_summary_roles_group_id_role_id_role_order_key
index info: 2048 total in 2 blocks; 904 free (0 chunks); 1144 used: group_summary_users_g_idx
index info: 2048 total in 2 blocks; 656 free (1 chunks); 1392 used: group_room_categories_group_id_category_id_key
index info: 3072 total in 2 blocks; 1048 free (1 chunks); 2024 used: group_summary_room_categories_group_id_category_id_cat_orde_key
index info: 3072 total in 2 blocks; 1080 free (1 chunks); 1992 used: group_summary_rooms_g_idx
index info: 3072 total in 2 blocks; 1064 free (1 chunks); 2008 used: group_summary_rooms_group_id_category_id_room_id_room_order_key
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: group_rooms_r_idx
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: group_rooms_g_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_stats_historical_end_ts
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: user_stats_historical_pkey
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: user_stats_current_pkey
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: current_state_delta_stream_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: device_federation_outbox_id
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: device_federation_outbox_destination_id
index info: 2048 total in 2 blocks; 416 free (1 chunks); 1632 used: device_lists_outbound_pokes_user
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: device_lists_outbound_pokes_stream
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: device_lists_outbound_pokes_id
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: cache_invalidation_stream_by_instance_id
index info: 2048 total in 2 blocks; 792 free (0 chunks); 1256 used: e2e_cross_signing_keys_stream_idx
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: e2e_cross_signing_keys_idx
index info: 2048 total in 2 blocks; 920 free (0 chunks); 1128 used: groups_idx
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: event_relations_relates
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: event_relations_id
index info: 2048 total in 2 blocks; 528 free (1 chunks); 1520 used: local_group_membership_u_idx
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: local_group_membership_g_idx
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: server_keys_json_uniqueness
index info: 2048 total in 2 blocks; 872 free (0 chunks); 1176 used: private_user_data_max_stream_id_lock_key
index info: 3072 total in 2 blocks; 696 free (1 chunks); 2376 used: event_txn_id_txn_id
index info: 2048 total in 2 blocks; 840 free (0 chunks); 1208 used: event_txn_id_ts
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: event_txn_id_event_id
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: stream_positions_idx
index info: 3072 total in 2 blocks; 696 free (1 chunks); 2376 used: e2e_room_keys_with_version_idx
index info: 3072 total in 2 blocks; 840 free (1 chunks); 2232 used: device_inbox_user_stream_id
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used: device_inbox_stream_id_user_id
index info: 2048 total in 2 blocks; 608 free (1 chunks); 1440 used: room_tag_revisions_uniqueness
index info: 4096 total in 3 blocks; 2184 free (2 chunks); 1912 used: room_memberships_user_room_forgotten
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: room_memberships_user_id
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: room_memberships_room_id
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: room_memberships_event_id_key
index info: 3544 total in 3 blocks; 432 free (0 chunks); 3112 used: remote_media_repository_thumbn_media_origin_id_width_height_met
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used: public_room_index
195 more child contexts containing 421096 total in 376 blocks; 131848 free (135 chunks); 289248 used
WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used
PrivateRefCount: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
MdSmgr: 16384 total in 2 blocks; 3032 free (6 chunks); 13352 used
LOCALLOCK hash: 32768 total in 3 blocks; 16824 free (8 chunks); 15944 used
Timezones: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
ErrorContext: 8192 total in 1 blocks; 7928 free (0 chunks); 264 used
Grand total: 6187280 bytes in 671 blocks; 2143936 free (325 chunks); 4043344 used
So, something in our instance of synapse makes it query sometihng from postgres that will never get released or accumulates over time, but I can't figure out what. Debugging help wanted, if there is no known things to do.
ADD: While rootissue is exactly synapse postgres growing, we also deploy following bridges on same server, should they bear anything in this equation:
mautrix-telegram
mautrix-whatsapp
mautrix-facebook
mautrix-signal
SmsMatrix bot
appservice-slack
appservice-discord