This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Error backfilling in room for 10 minutes after purging history #9864
Description
The error in the logs is:
2021-04-21 12:48:14,592 - synapse.handlers.federation - 1239 - ERROR - GET-2201 - Failed to backfill from localhost:8875 because Trying to persist state with unpersisted prev_group: 3369
Traceback (most recent call last):
File "/venv/lib/python3.7/site-packages/synapse/handlers/federation.py", line 1208, in try_backfill
dom, room_id, limit=100, extremities=extremities
File "/venv/lib/python3.7/site-packages/synapse/handlers/federation.py", line 1031, in backfill
context = await self.state_handler.compute_event_context(event)
File "/venv/lib/python3.7/site-packages/synapse/state/__init__.py", line 385, in compute_event_context
current_state_ids=state_ids_after_event,
File "/venv/lib/python3.7/site-packages/synapse/storage/state.py", line 592, in store_state_group
event_id, room_id, prev_group, delta_ids, current_state_ids
File "/venv/lib/python3.7/site-packages/synapse/storage/databases/state/store.py", line 492, in store_state_group
"store_state_group", _store_state_group_txn
File "/venv/lib/python3.7/site-packages/synapse/storage/database.py", line 667, in runInteraction
**kwargs,
File "/venv/lib/python3.7/site-packages/synapse/storage/database.py", line 743, in runWithConnection
self._db_pool.runWithConnection(inner_func, *args, **kwargs)
File "/venv/lib/python3.7/site-packages/twisted/python/threadpool.py", line 238, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/venv/lib/python3.7/site-packages/twisted/python/threadpool.py", line 255, in <lambda>
ctx, func, *args, **kw
File "/venv/lib/python3.7/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/venv/lib/python3.7/site-packages/twisted/python/context.py", line 83, in callWithContext
return func(*args, **kw)
File "/venv/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
compat.reraise(excValue, excTraceback)
File "/venv/lib/python3.7/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
return function(*args, **kwargs)
File "/venv/lib/python3.7/site-packages/twisted/python/compat.py", line 403, in reraise
raise exception.with_traceback(traceback)
File "/venv/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
result = func(conn, *args, **kw)
File "/venv/lib/python3.7/site-packages/synapse/storage/database.py", line 737, in inner_func
return func(db_conn, *args, **kwargs)
File "/venv/lib/python3.7/site-packages/synapse/storage/database.py", line 531, in new_transaction
r = func(cursor, *args, **kwargs)
File "/venv/lib/python3.7/site-packages/synapse/storage/databases/state/store.py", line 419, in _store_state_group_txn
% (prev_group,)
Exception: Trying to persist state with unpersisted prev_group: 3369
This happens when something pulls the state group into the cache just before we purge history and delete the state group, at which point we fail to invalidate the state cache. If we backfill the purged history again, the worker hits the cache and gets the old, deleted state group.
This is likely not much an issue outside of tests, as the state_cache expires after an hour.