Skip to content

vmagent kafka ingestion has low throughput at high latency networks #9249

@f41gh7

Description

@f41gh7

Describe the bug

Currently, vmagent data ingestion into kafka is limited by network latency. And if brokers respond with delay, overall throughput is limited by a slowest broker.

I propose to introduce a concurrency setting, as we already have for kafka consumer. It should have default value of 1 and could be increased if needed.

vmagent must start worker per concurrency setting and perform client side round-robin balancing.

To Reproduce

start kafka cluster with brokers at high latency network.

Version

any version of vmagent with kafka

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

No response

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions