2. Debezium exposes a KAFKA_VERSION environment variable that contains which version of Kafka the image is built against. We also use this in our integration tests; can we update this build file and/or our integration tests to read the correct Kafka version from this image? AFAICT we can't overwrite the environment variable itself without hardcoding a value (which seems distasteful), but we might consider reading it from the Kafka repo's gradle.properties file either during the build or in our integration tests.

Seems there is no way to set an ENV with command output. I saved it in the $KAFKA_HOME/kafka_version file so it can be read during tests.

Ugh, sorry--it looks like license checking doesn't work on raw Dockerfiles (see mathieucarbou/license-maven-plugin#204).

Can we add an exclude for the Dockerfile? Should look something like this (in pom.xml):

<exclude>**/Dockerfile</exclude>

Sure.

Also, just a thought--we may want to rename the Dockerfile or at least add some detail to the path for it, since people may reasonably see a file in docker/Dockerfile and think that that has to do with running kcctl instead of running Kafka Connect. Do you think it'd make sense to put it in a path like docker/connect/trunk/Dockerfile?

What do you think to put it in the test resource folder?

What do you think to put it in the test resource folder?

This is interesting... I think it depends on how we plan on using the Dockerfile in the future. If we need to access it using Java code at test runtime, then src/test/resources would be a great fit. If we access it manually before running integration tests with it locally, then it probably doesn't belong anywhere in the src directory. If we access it programmatically using, e.g., the dockerfile-maven-plugin, then that falls into a bit of a grey area, but IMO it's still better to leave out of the src directory.

Did you have thoughts on how to leverage this in the future? I have a draft commit ready that adds support for running integration tests against this image, but not for building it. If it's fast enough, we could add trunk to our CI matrix and start building the image as part of that.

Did you have thoughts on how to leverage this in the future?

I saw that there are test containers that uses the connect image from Debezium. Do you think we can use also this one here?

I have a draft commit ready that adds support for running integration tests against this image, but not for building it.

If you plan to test it using test containers maybe we can think to build it on the fly . WDYT?

If it's fast enough, we could add trunk to our CI matrix and start building the image as part of that.

It requires more or less 6 minutes.

I saw that there are test containers that uses the connect image from Debezium. Do you think we can use also this one here?

Yes, that's exactly what my draft commit does 😄

If you plan to test it using test containers maybe we can think to build it on the fly . WDYT?

Sounds great! Not sure how easy this would be though--we use the DebeziumContainer class for our testing, which subclasses GenericContainer and may make it more complicated to use the ImageFromDockerfile class from testcontainers. However, if we adopt this approach, it'd definitely make sense to place the Dockerfile in src/test/resources.

We don't have to block merging this PR on implementing build-on-the-fly logic; if you're confident that it can be done, we can merge this now (after moving the Dockerfile to a different directory) and tackle that later.

I saw that there are test containers that uses the connect image from Debezium. Do you think we can use also this one here?

Yes, that's exactly what my draft commit does 😄

If you plan to test it using test containers maybe we can think to build it on the fly . WDYT?

Sounds great! Not sure how easy this would be though--we use the DebeziumContainer class for our testing, which subclasses GenericContainer and may make it more complicated to use the ImageFromDockerfile class from testcontainers. However, if we adopt this approach, it'd definitely make sense to place the Dockerfile in src/test/resources.

We don't have to block merging this PR on implementing build-on-the-fly logic; if you're confident that it can be done, we can merge this now (after moving the Dockerfile to a different directory) and tackle that later.

I have found a way to build the image on the fly and use it as a DebeziumContainer. I did some tests and seems working, the first time it requires about 6 minutes to build the image but next tests execution will be more fast.

Not sure if the Dockerfile exclusion works after moving it on test resource folder. Let's see if CI pass.

Thanks @mfvitale, this is awesome!

Thank you!

1. Since the image build time is okay, we can start doing tests against trunk on our CI builds by adding TRUNK to the connect_version list.

I think it's reasonable.

2. If I want to test against trunk locally, the image will only be built once and then reused for subsequent test runs. This is great for cutting down test runtime, but may lead to using a stale image the next day, month, etc. We may want to add a cleaner way to force a rebuild of the image aside from manually deleting the debezium-connect:trunk (or debezium/connect:trunk) image.

Good point. I think that there isn't a solution that will safely save you from having a stale image. Two solution comes to my mind that can mitigate the risk:

• Put the deletion of the image by default and then add a property to force the reuse of the image (so the dev is conscious of what is doing).
• Leave the image deletion active by default and then delete it on Maven clean using docker-maven-plugin

Let me know what you think about these; happy to classify them both as follow-up items (and file issues for them) if you'd prefer not to address them in this PR.

Yes, better to have them on separate issues.

Great, thanks for all the excellent work @mfvitale!

Filed #381 and #382 as follow-ups.