Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

Prevent large ephemeral event traffic to appservices upon startup #10836

@anoadragon453

Description

@anoadragon453

@Half-Shot mentioned that upon start up, Synapse will send all missed ephemeral events to application services that have opted into receiving them (see MSC2409). There are some performance issues present with the feature's implementation in a couple of different circumstances.

Some background: a "stream token" is an incrementing integer. Every occurrence (a new typing event, a new read receipt, etc.) gets one of these IDs, and they're (usually rigidly) ordered in this way. Thus we can simply keep track of the last successfully sent stream token per app service to know what the AS needs to still receive.

  1. When the bridge is first added to the homeserver: it is assumed to have a last-successful stream token of 0:
    async def get_type_stream_id_for_appservice(
    self, service: ApplicationService, type: str
    ) -> int:
    if type not in ("read_receipt", "presence"):
    raise ValueError(
    "Expected type to be a valid application stream id type, got %s"
    % (type,)
    )
    def get_type_stream_id_for_appservice_txn(txn):
    stream_id_type = "%s_stream_id" % type
    txn.execute(
    # We do NOT want to escape `stream_id_type`.
    "SELECT %s FROM application_services_state WHERE as_id=?"
    % stream_id_type,
    (service.id,),
    )
    last_stream_id = txn.fetchone()
    if last_stream_id is None or last_stream_id[0] is None: # no row exists
    return 0
    else:
    return int(last_stream_id[0])
    return await self.db_pool.runInteraction(
    "get_type_stream_id_for_appservice", get_type_stream_id_for_appservice_txn
    )

When it comes time to send out the ephemeral events the appservice has missed, Synapse will send all read receipts and presence updates that it knows about and that are relevant to the namespaces the appservice has registered:

from_key = await self.store.get_type_stream_id_for_appservice(
service, "read_receipt"
)
receipts_source = self.event_sources.sources["receipt"]
receipts, _ = await receipts_source.get_new_events_as(
service=service, from_key=from_key
)
return receipts

This historical data isn't very useful to the appservice as it has just been registered.

  1. When a bridge has been offline for a while and comes back: it will receive all missed, relevant EDUs since it went offline. In particular, historical EDUs are typically less useful than the current state of things.

I'd be tempted to implement a time bound to the EDUs we send to appservices, such that very old EDUs are no longer considered.

Fixing both of these would help reduce the large amount of EDUs that can be sent to and overload both the appservice and Synapse itself when it tries to pull all of that information out of the database.

Finally, note that the above problems only seem to apply to the implementation of read receipts and presence events. Historical typing events are flat out not sent to appservices:

# We don't persist the token for typing_key for performance reasons

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-Application-ServiceRelated to AS supportT-EnhancementNew features, changes in functionality, improvements in performance, or user-facing enhancements.Z-Help-WantedWe know exactly how to fix this issue, and would be grateful for any contribution

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions