Skip to content

Cannot produce big messages #181

@turtledev1

Description

@turtledev1

I'm trying to produce a big kafka message in my tests, but I get a really weird error

... 57 55 2 18 115 101 99 116 105 111 110 52 57 12 115 101 97 116 49 55 24 115 101 97 116 82 101 102 52 57 57 57 56 2 18 115 101 99 116 105 111 110 52 57 12 115 101 97 116 49 56 24 115 101 97 116 82 101 102 52 57 57 57 57 2 18 115 101 99 116 105 111 110 52 57 12 115 101 97 116 49 57 24 115 101 97 116 82 101 102 53 48 48 48 48 2 18 115 101 99 116 105 111 110 52 57 0 0 0] [] {0 0 <nil>}} []})
        at github.com/mostafa/xk6-kafka.(*Kafka).writerClass.func1 (native)
        at insertion_simulation (file:///Users/marc-andre.bouvrette/Documents/Entry/k6-performance-tests/build/insertion-simulation.js:198:17(39))
        at native  executor=per-vu-iterations scenario=default source=stacktrace

Note: I didn't pasted the whole error because it's too big, but it's basically a bunch of numbers like these.

Here is my code

import { Connection, SchemaRegistry, SCHEMA_TYPE_AVRO, Writer } from 'k6/x/kafka';
import { config } from './config/config';
import { generateData } from './data';

const brokers = config.kafka.brokerList;
const schemaRegistry = new SchemaRegistry({
    url: config.kafka.schemaRegistryUrl
});
const topic = config.kafka.topic;

const writer = new Writer({
    brokers: brokers,
    topic: topic,
    compression: 'zstd',
    autoCreateTopic: true,
});

const connection = new Connection({
    address: brokers[0],
});

const valueSchemaObject = schemaRegistry.getSchema({
    subject: `${topic}-value`,
    schemaType: SCHEMA_TYPE_AVRO,
});

export function teardown() {
    writer.close();
    connection.close();
}

export default function () {
    for (let index = 0; index < 1; index++) {
        const messages = [
            {
                value: schemaRegistry.serialize({
                    data: generateData(), // a big JSON (around 30k items in an array). Works in my java project Also works with 27k items
                    schema: valueSchemaObject,
                    schemaType: SCHEMA_TYPE_AVRO,
                }),
            },
        ];
        writer.produce({ messages: messages });
    }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    ❓ QuestionFurther information is requested

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions