We'll be implementing a buffer using Kafka and the plugin server to ensure we associate events with the right distinct ID around the "identify edge", which we can also refer to as "conversion" events.

The solution will work as follows:

1. Add a new Kafka topic for the buffer, called e.g. events_buffer, where messages contain event payloads as well as an extra field process_at
2. When processing an event, run processEvent and onEvent normally. However, at the very "end" of the ingestEvent task, make a decision to send an event to ClickHouse directly or to the buffer topic. The heuristic for this is as follows:

anonymous events -> clickhouse
$identify -> clickhouse
non-anonymous events without a person -> buffer
non-anonymous events within conversion window -> buffer

3. Set up a new consumer on the main thread to consume from the buffer topic and send to a worker task doing the following:

• Look up person id
• Add id to event
• Produce event to Kafka topic consumed by CH

4. The consumer should work as follows:

   1. Pull a message from Kafka
   2. Check process_at. For t = process_at - now, if t > 0, don't commit the offset, finish the execution, stop the consumer and sleep for t. If t <= 0, ingest the event now

Won't waste a bunch of time making a graph that's a perfect representation of the world, but this should give a good overview of how this system will work:

This issue previously outlined a ClickHouse buffer solution that we've decided again. Click below to see its content.