This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Federation catchup for a destination can block indefinitely on un-partial stating #15220
Description
What we think happened
- A user faster joined a remote room
- Some events were sent into the room, but not transmitted successfully
- The user left the room before it became full state
- The federation catchup loop tried to transmit events in the room, but blocked on full state. This also blocked *all* catchup transmission for the destination.
Stack trace
File "/home/synapse/src/synapse/metrics/background_process_metrics.py", line 242, in run
return await func(*args, **kwargs)
File "/home/synapse/src/synapse/federation/sender/per_destination_queue.py", line 321, in _transaction_transmission_loop
await self._catch_up_transmission_loop()
File "/home/synapse/src/synapse/federation/sender/per_destination_queue.py", line 544, in _catch_up_transmission_loop
new_pdus = await filter_events_for_server(
File "/home/synapse/src/synapse/visibility.py", line 650, in filter_events_for_server
event_to_memberships = await _event_to_memberships(
File "/home/synapse/src/synapse/visibility.py", line 746, in _event_to_memberships
event_to_state_ids = await storage.state.get_state_ids_for_events(
File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
return await func(*args, **kwargs) # type: ignore[misc]
File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
return await func(*args, **kwargs) # type: ignore[misc]
File "/home/synapse/src/synapse/storage/controllers/state.py", line 267, in get_state_ids_for_events
event_to_groups = await self.get_state_group_for_events(
File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
return await func(*args, **kwargs) # type: ignore[misc]
File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
return await func(*args, **kwargs) # type: ignore[misc]
File "/home/synapse/src/synapse/storage/controllers/state.py", line 374, in get_state_group_for_events
await self._partial_state_events_tracker.await_full_state(event_ids)
File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
return await func(*args, **kwargs) # type: ignore[misc]
File "/home/synapse/src/synapse/storage/util/partial_state_events_tracker.py", line 82, in await_full_state
logger.info(