-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Closed
Description
I was making a project but thought it's best to leave each pair to decide how they want to organize their work. Nevertheless, I spent a bit of time mapping out some tasks for each of our goals so we can have a high-level overview of the team as a whole.
This list is certainly rough and non-exhaustive. Pairs will probably have a better sense of what to work on once they're in it.
Project Yeetcode (@tiina303 @Twixes)
Goal
Eliminate all legacy ingestion code from the plugin server.
Tasks:
- Remove all code that relies on Postgres for ingestion and other capabilities moved to CH (e.g. plugin log entries). Essentially get rid of the
if (this.kafkaProducer)
check - Update all tests accordingly so that we maintain/increase coverage. Currently a lot of stuff is tested only on the Postgres side (at least conceptually)
- Update public jobs to no longer use Celery: hitting the Graphile table on Postgres directly is probably best, as we'll skip the communication overhead. This should be straightforward but if for some reason it isn't, probably fine to use the events pipeline to achieve this.
- Remove all the Celery code from the plugin server.
Notes
- I'd highly recommend starting with the public jobs. Not because this is most important but because @macobo has already gotten started with some ingestion refactors related to the plugin server split goal, so working on public jobs should keep you busy for a little bit and help reduce conflicts between us all.
Project Fission (@macobo @yakkomajuri)
Goal
Have the plugin server split between ingestion and runner servers live on Cloud
Tasks:
- Map out the plugin server in terms of capabilities
- Run the event pipeline fully on the workers (test this on prod)
- Add configuration for both the runner and the ingestion server
- Make
onEvent
read from the CH ingestion topic (and ensure this change can be made safely) - Make
onAction
use the same event asonEvent
and run on the runner - Determine how to fit the buffer into the new architecture
- Devise a plan for the future of
processEvent
(extra)
Extra tasks
Pick these up if you're done with your goal quickly or are procrastinating etc.
- Determine how to handle $identify merges in conjunction with the buffer (@tiina303 & @yakkomajuri already on this)
- Understand how
kafkajs
behaves if one broker is down (and fix issues if there are any). Some context here. - Define plugin retries policy #8829
- Improvements to plugin jobs #7133
- Runbooks and alerts
tiina303