Skip to content

Limited YAML support while working with Kafka in Quarkus #49407

@ashish20790

Description

@ashish20790

Describe the bug

We were trying to create a consumerservice to capture CDC changes from Kafka. We are not using kafka-streams in our service as we are not interested in creating Topology.

We have defined our configurations in yaml files and due to this we were getting below error.

"org.apache.kafka.common.config.ConfigException","message":"Missing required configuration \"schema.registry.url\" which has no default value.

When we migrated our configurations to properties files, it started working correctly.

kafka:
  # Kafka broker configuration
  bootstrap:
    servers: localhost:9092

  # Producer configuration (for exactly-once semantics)
  enable:
    idempotence: true
  acks: all
  # Schema Registry configuration
  schema:
    registry:
      url: http://localhost:8081

mp:
  messaging:
    # Confluent Schema Registry configuration
    connector:
      smallrye-kafka:
        schema:
          registry:
            url: http://localhost:8081

    # Consumer configuration for your CDC topic
    incoming:
      readSegmentsAssociationValueEvent:
        connector: smallrye-kafka
        topic: nextgen_dev_000_segmentsassociation_raw_v1
        group:
          id: segments-association-consumer-group
        # Ignore deserialization failures
        fail-on-deserialization-failure: true
        # Confluent AVRO deserializers
        key:
          deserializer: org.apache.kafka.common.serialization.StringDeserializer
        value:
          deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
        # Confluent Schema Registry specific settings
        schema:
          registry:
            url: http://localhost:8081
        specific:
          avro:
            reader: true
        # Auto offset reset
        auto:
          offset:
            reset: earliest
          commit:
            interval:
              ms: 5000
        # Consumer performance tuning for CDC workloads
        max:
          poll:
            records: 100
        enable:
          auto:
            commit: true
        # Isolation level for exactly-once processing
        isolation:
          level: read_committed
        # Error handling - retry configuration
        failure-strategy: ignore
        batch: false
        # Disable both liveness and readiness checks with `health-enabled=false`:
#        health-enabled: false
        health-readiness-enabled: true

Expected behavior

Quarkus with Kafka should support both Yaml and properties configurations

Actual behavior

Quarkus with Kafka supports properties configurations as of now.

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions