-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Split different RDATA
streams into different redis channels #12461
Description
Similar to #12460, we should have a separate pub/sub channel for each RDATA
stream so that workers only need to subscribe to the channels they actually need.
This is a bit tricky to do safely, i.e. that workers do correctly subscribe to streams they need, especially for caching purposes. I think the best approach is instead of having the replication client call into the various handlers and storage classes:
synapse/synapse/replication/tcp/client.py
Lines 134 to 148 in e3a49f4
async def on_rdata( | |
self, stream_name: str, instance_name: str, token: int, rows: list | |
) -> None: | |
"""Called to handle a batch of replication data with a given stream token. | |
By default this just pokes the slave store. Can be overridden in subclasses to | |
handle more. | |
Args: | |
stream_name: name of the replication stream for this batch of rows | |
instance_name: the instance that wrote the rows. | |
token: stream token for this batch of rows | |
rows: a list of Stream.ROW_TYPE objects as returned by Stream.parse_row. | |
""" | |
self.store.process_replication_rows(stream_name, instance_name, token, rows) |
instead each class registers with the replication client in their __init__
each stream they care about with a callback to handle that particular streams rows. That way a worker only subscribes to a stream if a class that requires is actually instantiated.
The downside of this is that it's a bit magic, and pulling in a class can have the side effect of causing a stream to suddenly be subscribed to.