[KIP-1102] Enable clients to rebootstrap based on timeout or error code #4981

emasab · 2025-02-28T12:11:33Z

No description provided.

confluent-cla-assistant · 2025-02-28T12:11:42Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

Copilot

Pull Request Overview

This PR implements rebootstrap functionality for clients when metadata retrieval fails due to a timeout or a specific error code. Key changes include:

Adding a new special error action to trigger immediate rebootstrap in the metadata request error path.
Incrementing the metadata API version from 12 to 13 and updating all dependent components (mock handlers, configuration, and documentation).
Introducing a new configuration property and timer restart function to control rebootstrap timing.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/rdkafka_request.c	Updated error handling to invoke rebootstrap upon special error condition.
src/rdkafka_mock_handlers.c	Modified mock responses to support API version 13 and handle error conditions gracefully.
src/rdkafka_metadata.c	Added reading of top-level error codes and rebootstrap timer restart after metadata parse.
src/rdkafka_int.h & src/rdkafka.c	Introduced new macros and a timer restart function for rebootstrap.
src/rdkafka_conf.h & src/rdkafka_conf.c	Added a new configuration property for rebootstrap trigger timeout.
INTRODUCTION.md, CONFIGURATION.md, CHANGELOG.md	Updated documentation to reflect changes supporting rebootstrap.

pranavrth

First pass related to code.

CHANGELOG.md

pranavrth

Going through test cases.

pranavrth · 2025-06-17T07:44:35Z

CHANGELOG.md

 ## Enhancements and Fixes

+ * [KIP-899](https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap) Allow producer and consumer clients to rebootstrap


We should be upgrading the release notes as well. How should we do that?

You mean the release in GitHub? I can modify it after merging

CHANGELOG.md

src/rdkafka_metadata.c

pranavrth · 2025-06-17T10:51:37Z

src/rdkafka_mock_handlers.c

-                if (rkbuf->rkbuf_reqhdr.ApiVersion >= 1) {
-                        /* Response: Brokers.Rack (Matt's going to love this) */
-                        rd_kafka_buf_write_str(resp, mrkb->rack, -1);
+        if (!err) {


Put every things in !err if condition instead of adding !err to every conditions.

pranavrth · 2025-06-17T10:52:42Z

src/rdkafka_mock_handlers.c

@@ -1378,7 +1386,7 @@ static int rd_kafka_mock_handle_Metadata(rd_kafka_mock_connection_t *mconn,
                                       &IncludeTopicAuthorizedOperations);
        }

-        if (list_all_topics) {
+        if (!err && list_all_topics) {


pranavrth · 2025-06-17T10:52:47Z

src/rdkafka_mock_handlers.c

@@ -1389,7 +1397,7 @@ static int rd_kafka_mock_handle_Metadata(rd_kafka_mock_connection_t *mconn,
                            mtopic->id, mtopic->name, mtopic, mtopic->err);
                }

-        } else if (requested_topics) {
+        } else if (!err && requested_topics) {


src/rdkafka_mock_handlers.c

src/rdkafka.c

pranavrth

Minor nits.

CHANGELOG.md

included the `metadata.recovery.strategy=none` in the cases that can generate a fatal error. Mock cluster Metadata RPC upgrade to v13 and support for the forcing a top-level error code

string as bootstrap servers or with both bootstrap servers and manually added brokers

metadata.recovery.strategy=none and don't re-bootstrap at all in that case

pranavrth

LGTM!.

pranavrth · 2025-06-18T21:36:26Z

Thanks @emasab

airlock-confluentinc bot force-pushed the dev_kip899 branch from 5bff66d to 313f6a0 Compare March 10, 2025 17:08

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 2c542b0 to b3c8716 Compare March 10, 2025 17:10

pranavrth mentioned this pull request Mar 13, 2025

[KIP-899] Allow producer and consumer clients to rebootstrap #4980

Merged

airlock-confluentinc bot force-pushed the dev_kip899 branch from 313f6a0 to b845ac2 Compare March 13, 2025 11:41

airlock-confluentinc bot force-pushed the dev_kip1102 branch from b3c8716 to d072e63 Compare March 13, 2025 11:42

airlock-confluentinc bot force-pushed the dev_kip899 branch from b845ac2 to aeda39e Compare March 18, 2025 17:58

airlock-confluentinc bot force-pushed the dev_kip1102 branch from d072e63 to 2de18f3 Compare March 18, 2025 18:00

airlock-confluentinc bot force-pushed the dev_kip899 branch from aeda39e to 620aa32 Compare March 25, 2025 17:47

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 2de18f3 to 02733a7 Compare March 25, 2025 17:47

airlock-confluentinc bot force-pushed the dev_kip899 branch from 1428df0 to ee212c1 Compare March 28, 2025 09:51

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 02733a7 to c40e9c6 Compare March 28, 2025 09:52

airlock-confluentinc bot force-pushed the dev_kip899 branch 5 times, most recently from f99417f to 1e51911 Compare March 28, 2025 16:02

airlock-confluentinc bot force-pushed the dev_kip1102 branch 2 times, most recently from f294613 to 3eb055f Compare March 31, 2025 09:33

airlock-confluentinc bot force-pushed the dev_kip899 branch 2 times, most recently from a64cd5b to 7ff294e Compare March 31, 2025 10:05

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 3eb055f to 86fd198 Compare March 31, 2025 10:08

airlock-confluentinc bot force-pushed the dev_kip899 branch from 7ff294e to 8762319 Compare March 31, 2025 16:54

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 86fd198 to 5c90461 Compare March 31, 2025 16:54

airlock-confluentinc bot force-pushed the dev_kip899 branch from 8762319 to 6dbeae7 Compare March 31, 2025 19:26

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 5c90461 to 853b478 Compare March 31, 2025 19:27

airlock-confluentinc bot force-pushed the dev_kip899 branch from 6dbeae7 to 0aede38 Compare April 1, 2025 14:00

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 853b478 to 4c12f61 Compare April 1, 2025 14:01

airlock-confluentinc bot force-pushed the dev_kip899 branch 2 times, most recently from e9889e5 to 919a9bc Compare April 1, 2025 17:52

emasab requested a review from a team as a code owner April 24, 2025 14:57

Copilot AI reviewed Apr 24, 2025

View reviewed changes

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 8a94873 to 468a55e Compare April 24, 2025 15:07

airlock-confluentinc bot force-pushed the dev_kip1102 branch from c97b56a to 29e814e Compare May 26, 2025 09:48

pranavrth requested changes May 27, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

airlock-confluentinc bot force-pushed the dev_kip1102 branch 2 times, most recently from f2728b3 to 9e6661d Compare June 13, 2025 07:43

pranavrth requested changes Jun 17, 2025

View reviewed changes

pranavrth requested changes Jun 18, 2025

View reviewed changes

src/rdkafka.c Show resolved Hide resolved

src/rdkafka.c Show resolved Hide resolved

airlock-confluentinc bot force-pushed the dev_kip1102 branch from f96b7a2 to 3f6e2fa Compare June 18, 2025 19:20

pranavrth requested changes Jun 18, 2025

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

emasab added 14 commits June 18, 2025 23:01

[KIP-1102] Enable clients to rebootstrap based on timeout or error code

e6e4a92

Completed implementation and tests

fea0e80

included the `metadata.recovery.strategy=none` in the cases that can generate a fatal error. Mock cluster Metadata RPC upgrade to v13 and support for the forcing a top-level error code

CHANGELOG entry

3f25ddd

Style fix

d30ea07

Test tuning for macOS

f4b4b4c

Fix zero array literal

88e558b

Add additional re-bootstrap tests with an empty

24b40d9

string as bootstrap servers or with both bootstrap servers and manually added brokers

Doc about purge-brokers tests

f55f106

Remove the fatal error code on

05b1c87

metadata.recovery.strategy=none and don't re-bootstrap at all in that case

Fix test 0152

34d45aa

Implementation comments

981f6ff

Tests related comments

2a588a7

Revert tests for moving them to a later PR

1f27a5e

Changelog entry comments

f94ffd2

airlock-confluentinc bot force-pushed the dev_kip1102 branch from 78dd3b4 to f94ffd2 Compare June 18, 2025 21:03

pranavrth approved these changes Jun 18, 2025

View reviewed changes

emasab merged commit 3137946 into master Jun 18, 2025
2 checks passed

emasab deleted the dev_kip1102 branch June 18, 2025 21:37

		## Enhancements and Fixes

		* [KIP-899](https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap) Allow producer and consumer clients to rebootstrap

[KIP-1102] Enable clients to rebootstrap based on timeout or error code #4981

[KIP-1102] Enable clients to rebootstrap based on timeout or error code #4981

Uh oh!

Conversation

emasab commented Feb 28, 2025

Uh oh!

confluent-cla-assistant bot commented Feb 28, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

pranavrth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pranavrth left a comment

Choose a reason for hiding this comment

Uh oh!

pranavrth Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

emasab Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

pranavrth Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pranavrth Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

pranavrth Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

pranavrth Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pranavrth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pranavrth left a comment

Choose a reason for hiding this comment

Uh oh!

pranavrth commented Jun 18, 2025

Uh oh!

Uh oh!

Uh oh!