Skip to content

Nuke protobuf #9207

@yakkomajuri

Description

@yakkomajuri

The persons on events changes we're about to start making require sending new bits of data over through Kafka. These would require changes in the protobuf schema but honestly we might be better off just dropping it at this stage.

@macobo has done some on the initial legwork to support non-protobuf messages on #8333 but I suggest we make the move now.

This would require:

  1. Setting up a new Kafka topic (manually on Cloud, self-hosted topics are auto-created)
  2. Setting up a new set of Kafka + MV tables to consume from the topic into writable_events
  3. Begin sending all new events in a JSON encoded format to the new JSON topic
  4. Keep the protobuf topic and tables around for a release and then nuke it on the following one, so we ensure all events from that path are correctly processed
  5. Remove all the related code

A gotcha here is at step 2. What if the user isn't sharded but becomes sharded during this transition? We probably need to retroactively update the sharding async migration to drop and recreate the MV for the JSON topic as well in order to do this so that we write to the correct table.

Metadata

Metadata

Assignees

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions