Skip to content

Ingestion Sprint 1.36.0 2/2 Plan #9763

@yakkomajuri

Description

@yakkomajuri

I was making a project but thought it's best to leave each pair to decide how they want to organize their work. Nevertheless, I spent a bit of time mapping out some tasks for each of our goals so we can have a high-level overview of the team as a whole.

This list is certainly rough and non-exhaustive. Pairs will probably have a better sense of what to work on once they're in it.

Project Yeetcode (@tiina303 @Twixes)

Goal

Eliminate all legacy ingestion code from the plugin server.

Tasks:

  • Remove all code that relies on Postgres for ingestion and other capabilities moved to CH (e.g. plugin log entries). Essentially get rid of the if (this.kafkaProducer) check
  • Update all tests accordingly so that we maintain/increase coverage. Currently a lot of stuff is tested only on the Postgres side (at least conceptually)
  • Update public jobs to no longer use Celery: hitting the Graphile table on Postgres directly is probably best, as we'll skip the communication overhead. This should be straightforward but if for some reason it isn't, probably fine to use the events pipeline to achieve this.
  • Remove all the Celery code from the plugin server.

Notes

  • I'd highly recommend starting with the public jobs. Not because this is most important but because @macobo has already gotten started with some ingestion refactors related to the plugin server split goal, so working on public jobs should keep you busy for a little bit and help reduce conflicts between us all.

Project Fission (@macobo @yakkomajuri)

Goal

Have the plugin server split between ingestion and runner servers live on Cloud

Tasks:

  • Map out the plugin server in terms of capabilities
  • Run the event pipeline fully on the workers (test this on prod)
  • Add configuration for both the runner and the ingestion server
  • Make onEvent read from the CH ingestion topic (and ensure this change can be made safely)
  • Make onAction use the same event as onEvent and run on the runner
  • Determine how to fit the buffer into the new architecture
  • Devise a plan for the future of processEvent (extra)

Extra tasks

Pick these up if you're done with your goal quickly or are procrastinating etc.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions