-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Closed
Description
The persons on events changes we're about to start making require sending new bits of data over through Kafka. These would require changes in the protobuf schema but honestly we might be better off just dropping it at this stage.
@macobo has done some on the initial legwork to support non-protobuf messages on #8333 but I suggest we make the move now.
This would require:
- Setting up a new Kafka topic (manually on Cloud, self-hosted topics are auto-created)
- Setting up a new set of Kafka + MV tables to consume from the topic into
writable_events
- Begin sending all new events in a JSON encoded format to the new JSON topic
- Keep the protobuf topic and tables around for a release and then nuke it on the following one, so we ensure all events from that path are correctly processed
- Remove all the related code
A gotcha here is at step 2. What if the user isn't sharded but becomes sharded during this transition? We probably need to retroactively update the sharding async migration to drop and recreate the MV for the JSON topic as well in order to do this so that we write to the correct table.
Metadata
Metadata
Assignees
Type
Projects
Status
Done