fix(clickhouse): Add auto-retry logic on SSL reset error #3733
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
floganz
approved these changes
May 27, 2025
vincent-pochet
added a commit
that referenced
this pull request
May 27, 2025
## Description Similarly to #3733, this PR adds auto-retry for `ActiveRecord::ActiveRecordError` on Clickhouse queries
vincent-pochet
added a commit
that referenced
this pull request
May 27, 2025
vincent-pochet
added a commit
that referenced
this pull request
Jun 20, 2025
## Context This PR follows #3733 but we are still seeing some `Errno::ECONNRESET Connection reset by peer - SSL_connect` ## Description It refactors a bit the way we handle retries as it appears that some `events.to_sql` are also hiting the database. NOTE: this is a quick fix, a future PR will had some logic to rewrite theses queries in pure SQL to reduce this unintended queries that generates some load on the clickhouse instance
diegocharles
pushed a commit
that referenced
this pull request
Jul 11, 2025
## Context This PR follows #3733 but we are still seeing some `Errno::ECONNRESET Connection reset by peer - SSL_connect` ## Description It refactors a bit the way we handle retries as it appears that some `events.to_sql` are also hiting the database. NOTE: this is a quick fix, a future PR will had some logic to rewrite theses queries in pure SQL to reduce this unintended queries that generates some load on the clickhouse instance
vincent-pochet
added a commit
that referenced
this pull request
Aug 19, 2025
## Context This PR follows #3733 ## Description In some situation, a "NoMethodError: undefined method 'select_all' for nil" is raised when running `::Clickhouse::EventsEnriched.connection.select_all`. This is an issue with the Active record connector for Clickhouse. This PR just handle an automatic retry when the error is raised
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Some time, with Clickhouse queries, a
Errno::ECONNRESET Connection reset by peer - SSL_connectis raised.This happen when the Clickhouse cluster is under eavy load and is scaled automatically.
Description
Since these error are intermediary, this PR adds a auto-retry system (up to 3 retries) to reduce the number of errors in the following processes (billing, current usage, daily usage...)
NOTE: we should check how to address this at the Clickhouse connector level, but it is not currently possible to pass an option to enable
max_retrieson the underlying Net::HTTP client