Proposed change

In #276 it became theoretically possible to use an external clickhouse cluster.

In practice though, using the cluster requires quite a few fixes to land in a release of PostHog which will take ~month.

This issue documents these fixes and how to get everything working.

Required PRs

• Make service requirements less strict for clickhouse posthog#8311
• Update clickhouse-orm posthog#8314 / Support altinity cloud when fetching server version infi.clickhouse_orm#7
• Allow overriding kafka host for clickhouse via KAFKA_URL_CLICKHOUSE env var posthog#8332
• Allow disabling protobuf with CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS env posthog#8333

Also relevant: PostHog/posthog#8334

Testing the changes and sample values.yaml

I tested out a build where all the above fixes were baked in. The procedure was:

• ngrok tcp 9092 to expose my clickhouse installation to the world. Note down the exposed url and replace it within the values file below
• helm install -f altinity-cloud-values.yaml --timeout 20m --create-namespace --namespace posthog posthog charts/posthog/ --atomic --wait --wait-for-jobs --debug
• kubectl port-forward service/posthog-posthog-kafka 9092:9092 -n posthog while helm is running to expose kafka

The values file required some annoying overrides:

cloud: "private"

image:
  repository: XXX
  sha: XXX

env:
- name: KAFKA_URL_FOR_CLICKHOUSE
  value: "kafka://4.tcp.ngrok.io:10912"
- name: CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS
  value: "1"

kafka:
  advertisedListeners: "INTERNAL://$(MY_POD_NAME).posthog-posthog-kafka-headless.posthog.svc.cluster.local:9093,CLIENT://4.tcp.ngrok.io:10912"

clickhouse:
  enabled: false

externalClickhouse:
  host: xxx.demo.altinity.cloud
  user: xxx
  password: "xxx"
  cluster: posthog
  secure: true

Notes for the future

We should figure out a more stream-lined way of exposing kafka here - there's a bit of a catch-22 going on with advertisedListeners and migrations - we need to know kafka external hostname by the time of migrations.

Alternatives

As an alternative to the protobuf change, users could also manually upload the protobuf file by:

1. Opening the cluster, clicking Configure > Settings
2. Upload these two settings

Note that this needs to be done prior to first helm install and events.proto is checked into our current codebase.

The rest of linked PRs still remain required though.