Re-factor fluentd CI workflows #1472

joshuabaird · 2025-02-17T13:50:37Z

This PR re-factors the CI workflows for building and publishing fluent-operator's fluentd image.

Why the re-factor?

The fluentd image had not been touched for quite some time and there was a lot of old cruft (old Dockerfiles, etc)
The old build process was using QEMU for building arm64 images on amd64 and it was prone to issues (seg faults in modern gcc stacks) and was very slow (20+ minutes to build an image)
We were publishing a "base" image which was not actually being used
We were building the fluentd image from "scratch" instead of basing our image off of the upstream fluent/fluentd image and there was no benefit to doing so
The build process was actually broken for new (v1.17.1+) images of fluentd

The re-factor

Removes unused/legacy Dockerfiles and uses a single Dockerfile for both amd64 and arm64 images
The fluentd image is now based from the upstream fluent/fluentd image which makes it easier to maintain
Builds both amd64 and arm64 images
Builds arm64 images on native arm64 runners instead of using QEMU which makes it much faster (~3min vs 20+ min) and simplifies our Dockerfile
Uses Debian-based upstream fluentd images so that we get standard jmellaoc support
Replaces the clone-docker-image-action CI workflow with native functionality that publishes multi-arch images/manifests to both GHCR and Docker Hub
Temporarily removes the "image scan" workflow since we were publishing images regardless of the result
Adopts the VERSION file approach that was implemented for fluent-bit in feat: add VERSION file for fluentbit image #1447

Future enhancements

We can re-factor the fluent-bit CI to follow this same pattern and perhaps we can use a single workflow to build both fluent-bit and fluentd images.

Signed-off-by: Josh Baird <jbaird@galileo.io>

joshuabaird · 2025-02-18T16:25:10Z

cc: @benjaminhuo, @cw-Guo and @wenchajun for review.

Once merged, we'll publish a v1.17.1 (and maybe v1.18.0) image of fluentd.

cw-Guo · 2025-02-19T00:35:17Z

cmd/fluent-watcher/fluentd/Dockerfile

+
+LABEL org.opencontainers.image.description "A Fluentd image for use with fluent-operator"
+
+USER root


I see some of the previous Dockerfiles run as non-root user, for example: https://github.com/fluent/fluent-operator/pull/1472/files#diff-57700540a4a9a3e075df0d7035170b3d3e4559e3a9e9fcef66ac0996f85447ebL61-L66.

Not sure whether we want to implement this too in the new dockerfile. From my understanding, it will enhance the security.

@cw-Guo i would welcome the images to run as non-root, this is a Best Practice

Good callout -- I'll get this fixed up to run as a non-root user. This was actually an oversight.

This is now fixed -- confirmed that the process is running as the fluent user:

$ USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND fluent 1 0.0 0.0 1226248 2840 ? Ssl 20:43 0:00 /fluentd/bin/fluentd-watcher fluent 13 5.1 0.6 172192 50512 ? Sl 20:43 0:00 /usr/local/bin/ruby /usr/local/bundle/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins

Signed-off-by: Josh Baird <jbaird@galileo.io>

joshuabaird · 2025-02-19T20:51:18Z

cmd/fluent-watcher/fluentd/main.go

@@ -20,7 +20,7 @@ import (
 )

 const (
-	defaultBinPath      = "/usr/bin/fluentd"
+	defaultBinPath      = "/usr/local/bundle/bin/fluentd"


The default path for the fluentd binary (both amd64 and arm64) is /usr/local/bundle/bin in the upstream images (which we are now using).

joshuabaird · 2025-02-21T12:47:36Z

@benjaminhuo @wenchajun Any issues with this? I would like to get it merged so we can publish new fluentd images for #1471.

* Bump Dockerfile to v1.17.1. Signed-off-by: Josh Baird <jbaird@galileo.io> * Update Dockerfile. Signed-off-by: Josh Baird <jbaird@galileo.io> * Try bumping bigdecimal. Signed-off-by: Josh Baird <jbaird@galileo.io> * Re-factor arm64 Dockerfile. Signed-off-by: Josh Baird <jbaird@galileo.io> * Include mime types. Signed-off-by: Josh Baird <jbaird@galileo.io> * Do not build detect-exceptions. Signed-off-by: Josh Baird <jbaird@galileo.io> * Simplify fluentd build pipeline. Signed-off-by: Josh Baird <jbaird@galileo.io> * Simplify fluentd build pipeline. Signed-off-by: Josh Baird <jbaird@galileo.io> * Simplify fluentd build pipeline. Signed-off-by: Josh Baird <jbaird@galileo.io> * Simplify fluentd build pipeline. Signed-off-by: Josh Baird <jbaird@galileo.io> * Rename some things. Signed-off-by: Josh Baird <jbaird@galileo.io> * Hm. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add arm64. Signed-off-by: Josh Baird <jbaird@galileo.io> * Try new workflow. Signed-off-by: Josh Baird <jbaird@galileo.io> * Try new workflow. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Produce a manifest. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add support for Docker Hub. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add support for Docker Hub. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add support for Docker Hub. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add support for Docker Hub. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add support for Docker Hub. Signed-off-by: Josh Baird <jbaird@galileo.io> * Prep workflow for prod. Signed-off-by: Josh Baird <jbaird@galileo.io> * Remove old Dockerfiles. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add notes. Signed-off-by: Josh Baird <jbaird@galileo.io> * Reorg some things. Signed-off-by: Josh Baird <jbaird@galileo.io> * Update BUILD docs. Signed-off-by: Josh Baird <jbaird@galileo.io> * Cleanup. Signed-off-by: Josh Baird <jbaird@galileo.io> * Use VERSION file. Signed-off-by: Josh Baird <jbaird@galileo.io> * Use VERSION file. Signed-off-by: Josh Baird <jbaird@galileo.io> * Oops. Signed-off-by: Josh Baird <jbaird@galileo.io> * Modify VERSION. Signed-off-by: Josh Baird <jbaird@galileo.io> * Modify VERSION. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add ARG. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add ARG. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add ARG. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add ARG. Signed-off-by: Josh Baird <jbaird@galileo.io> * Add ARG. Signed-off-by: Josh Baird <jbaird@galileo.io> * Update README. Signed-off-by: Josh Baird <jbaird@galileo.io> * Update README. Signed-off-by: Josh Baird <jbaird@galileo.io> * Update README. Signed-off-by: Josh Baird <jbaird@galileo.io> * Reset. Signed-off-by: Josh Baird <jbaird@galileo.io> * Use fluent user. Signed-off-by: Josh Baird <jbaird@galileo.io> * Tets. Signed-off-by: Josh Baird <jbaird@galileo.io> * Test. Signed-off-by: Josh Baird <jbaird@galileo.io> * Set amd64 fluentd path. Signed-off-by: Josh Baird <jbaird@galileo.io> * One more try. Signed-off-by: Josh Baird <jbaird@galileo.io> * This is it. Signed-off-by: Josh Baird <jbaird@galileo.io> * Reset. Signed-off-by: Josh Baird <jbaird@galileo.io> --------- Signed-off-by: Josh Baird <jbaird@galileo.io>

joshuabaird added 30 commits February 17, 2025 08:49

Bump Dockerfile to v1.17.1.

99a76c6

Signed-off-by: Josh Baird <jbaird@galileo.io>

Update Dockerfile.

6765de4

Signed-off-by: Josh Baird <jbaird@galileo.io>

Try bumping bigdecimal.

a566472

Signed-off-by: Josh Baird <jbaird@galileo.io>

Re-factor arm64 Dockerfile.

f0371d7

Signed-off-by: Josh Baird <jbaird@galileo.io>

Include mime types.

622ca61

Signed-off-by: Josh Baird <jbaird@galileo.io>

Do not build detect-exceptions.

59fcec3

Signed-off-by: Josh Baird <jbaird@galileo.io>

Simplify fluentd build pipeline.

92c3eae

Signed-off-by: Josh Baird <jbaird@galileo.io>

Simplify fluentd build pipeline.

03777d9

Signed-off-by: Josh Baird <jbaird@galileo.io>

Simplify fluentd build pipeline.

8423d81

Signed-off-by: Josh Baird <jbaird@galileo.io>

Simplify fluentd build pipeline.

0fb5c36

Signed-off-by: Josh Baird <jbaird@galileo.io>

Rename some things.

5b212bc

Signed-off-by: Josh Baird <jbaird@galileo.io>

Hm.

d969153

Signed-off-by: Josh Baird <jbaird@galileo.io>

Add arm64.

4cfe7be

Signed-off-by: Josh Baird <jbaird@galileo.io>

Try new workflow.

8399ba8

Signed-off-by: Josh Baird <jbaird@galileo.io>

Try new workflow.

7d85b78

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

71e8203

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

42d2833

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

b42c76e

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

b920b18

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

ceef7a7

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

88323e6

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

cb27856

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

fa0115e

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

0028000

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

fa9183c

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

bf2a0b6

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

11292ae

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

05d118b

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

371680f

Signed-off-by: Josh Baird <jbaird@galileo.io>

Produce a manifest.

59faeee

Signed-off-by: Josh Baird <jbaird@galileo.io>

joshuabaird added 12 commits February 18, 2025 10:24

Oops.

a084c41

Signed-off-by: Josh Baird <jbaird@galileo.io>

Modify VERSION.

0e3d4b3

Signed-off-by: Josh Baird <jbaird@galileo.io>

Modify VERSION.

cfccd34

Signed-off-by: Josh Baird <jbaird@galileo.io>

Add ARG.

6df397d

Signed-off-by: Josh Baird <jbaird@galileo.io>

Add ARG.

866270f

Signed-off-by: Josh Baird <jbaird@galileo.io>

Add ARG.

bac3932

Signed-off-by: Josh Baird <jbaird@galileo.io>

Add ARG.

3c42ee1

Signed-off-by: Josh Baird <jbaird@galileo.io>

Add ARG.

b18a9a5

Signed-off-by: Josh Baird <jbaird@galileo.io>

Update README.

0a71ceb

Signed-off-by: Josh Baird <jbaird@galileo.io>

Update README.

c8832cf

Signed-off-by: Josh Baird <jbaird@galileo.io>

Update README.

2b92b82

Signed-off-by: Josh Baird <jbaird@galileo.io>

Reset.

197faa0

Signed-off-by: Josh Baird <jbaird@galileo.io>

joshuabaird marked this pull request as ready for review February 18, 2025 16:25

joshuabaird requested a review from cw-Guo February 18, 2025 16:27

cw-Guo reviewed Feb 19, 2025

View reviewed changes

joshuabaird added 7 commits February 19, 2025 14:24

Use fluent user.

9059959

Signed-off-by: Josh Baird <jbaird@galileo.io>

Tets.

53370e8

Signed-off-by: Josh Baird <jbaird@galileo.io>

Test.

27d7ea9

Signed-off-by: Josh Baird <jbaird@galileo.io>

Set amd64 fluentd path.

7d8844b

Signed-off-by: Josh Baird <jbaird@galileo.io>

One more try.

f82a34c

Signed-off-by: Josh Baird <jbaird@galileo.io>

This is it.

be6b6ed

Signed-off-by: Josh Baird <jbaird@galileo.io>

Reset.

35c5494

Signed-off-by: Josh Baird <jbaird@galileo.io>

joshuabaird commented Feb 19, 2025

View reviewed changes

cw-Guo approved these changes Feb 21, 2025

View reviewed changes

joshuabaird merged commit 3f34713 into fluent:master Feb 21, 2025
6 checks passed

joshuabaird mentioned this pull request Mar 20, 2025

Re-factor fluent-bit image build CI workflows #1526

Closed

joshuabaird mentioned this pull request Mar 24, 2025

Re-factors CI workflow for building & publishing fluent-bit image #1531

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Re-factor fluentd CI workflows #1472

Re-factor fluentd CI workflows #1472

Uh oh!

joshuabaird commented Feb 17, 2025 •

edited

Loading

Uh oh!

joshuabaird commented Feb 18, 2025

Uh oh!

cw-Guo Feb 19, 2025

Uh oh!

localleon Feb 19, 2025

Uh oh!

joshuabaird Feb 19, 2025 •

edited

Loading

Uh oh!

joshuabaird Feb 19, 2025

Uh oh!

joshuabaird Feb 19, 2025

Uh oh!

joshuabaird commented Feb 21, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!


		LABEL org.opencontainers.image.description "A Fluentd image for use with fluent-operator"

		USER root

Re-factor fluentd CI workflows #1472

Re-factor fluentd CI workflows #1472

Uh oh!

Conversation

joshuabaird commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why the re-factor?

The re-factor

Future enhancements

Uh oh!

joshuabaird commented Feb 18, 2025

Uh oh!

cw-Guo Feb 19, 2025

Choose a reason for hiding this comment

Uh oh!

localleon Feb 19, 2025

Choose a reason for hiding this comment

Uh oh!

joshuabaird Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshuabaird Feb 19, 2025

Choose a reason for hiding this comment

Uh oh!

joshuabaird Feb 19, 2025

Choose a reason for hiding this comment

Uh oh!

joshuabaird commented Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshuabaird commented Feb 17, 2025 •

edited

Loading

joshuabaird Feb 19, 2025 •

edited

Loading

joshuabaird commented Feb 21, 2025 •

edited

Loading