Skip to content

Allow disabling protobuf with CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS env #8333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 1, 2022
Merged

Allow disabling protobuf with CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS env #8333

merged 3 commits into from
Feb 1, 2022

Conversation

macobo
Copy link
Contributor

@macobo macobo commented Jan 28, 2022 •
edited
Loading

This makes integrating with external clickhouse providers (like altinity
cloud) possible as users don't need to use undocumented APIs to
upload the schemas.

It's pretty certain that events proto will also change over time so this avoids issues during upgrades as well.

Documenting for posterity, the alternative is to:

  1. Go to Configure > Settings
  2. Upload these two settings
    image

(events.proto is checked into our current codebase)

Related issues:

Note for reviewers:

  • I didn't figure out a good way to test this from the plugin-server side since it relies on a non-default env var being set in one of the tests during schema setup. This has however been tested on a live cluster. Advice welcome.

@macobo macobo mentioned this pull request Jan 28, 2022
Closed
@macobo macobo force-pushed the clickhouse-kafka-disable-protobuf branch 2 times, most recently from a283ef1 to 646a123 Compare January 28, 2022 13:35
@macobo macobo mentioned this pull request Jan 31, 2022
Closed
Base automatically changed from clickhouse-kafka-hosts to master January 31, 2022 08:21
macobo added 3 commits January 31, 2022 10:22
@macobo
Allow disabling protobuf with CLICKHOUSE_DISABLE_EXTERNAL_SCHEMAS env…
6e1fe55 
… var

This makes integrating with external clickhouse providers (like altinity
cloud) much easier as users don't need to use undocumented APIs to
upload the schemas.
@macobo
Keep underscores in protobuf. This allows using this payload for json…
262fb16 
… ingestion
@macobo
Timestamp format
f8eac95
@macobo macobo force-pushed the clickhouse-kafka-disable-protobuf branch from d19b249 to f8eac95 Compare January 31, 2022 08:22
@macobo macobo mentioned this pull request Jan 31, 2022
Merged
10 tasks
@macobo macobo marked this pull request as ready for review January 31, 2022 08:43
guidoiaquinti
guidoiaquinti approved these changes Jan 31, 2022
View reviewed changes
Copy link
Contributor

@guidoiaquinti guidoiaquinti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but I would appreciate as well @yakkomajuri 👀

yakkomajuri
yakkomajuri approved these changes Feb 1, 2022
View reviewed changes
Copy link
Contributor

@yakkomajuri yakkomajuri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for doing this

@macobo macobo merged commit 2e6aae7 into master Feb 1, 2022
@macobo macobo deleted the clickhouse-kafka-disable-protobuf branch February 1, 2022 22:30
@fuziontech
Copy link
Member

Nice!

@yakkomajuri yakkomajuri mentioned this pull request Mar 23, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
2 more reviewers

@guidoiaquinti guidoiaquinti guidoiaquinti approved these changes

@yakkomajuri yakkomajuri yakkomajuri approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@macobo @fuziontech @guidoiaquinti @yakkomajuri