Docker Multi Arch Builds Fix #1414

activenode · 2023-07-28T18:52:23Z

Okay so here's a few things that have changed but I can confirm it works and it builds because I did both of it.

Here's the ready-built stuff on my HUB: https://hub.docker.com/r/activenode/listmonk/tags
And I also deployed it on my Server to test it, including the new flags. Feel free to do the same.

There are 3 things essentially done in this PR:

Fixing the original Multi-Arch PR
Making listmonk conveniently work with docker stacks and documenting this
Improving UX Feedback on the SMTP settings page

I will also add comments to my own PR so you can follow up what is what. Feel free to merge whenever you want. Until then my own pipeline will still be running and pubbing docker images at activenode/listmonk.

Considering Multi-Arch:

QEMU missing

So when this was merged: #1344 it was a very good intent and good prework but I think the author didn't test this in GitHub Actions because that's where it definitely fails. It needs QEMU to even work properly.

Multi-Arch Problems 1

Multi-Arch is more than just building on different platforms. Different platforms have different syscalls. And that's where the whole build crashed because Utsname is kinda architecture specific. Since it wasn't a biggie, I just added a more universal solution which works fine.

Multi-Arch Problems 2

It gets even more complex and harder to maintain if we go beyond Linux. I mean I've seen the netbsd building fine but then again with Windows and darwin you open up a different system to a different CPU architecture.
I left it there, commented, for the reason of clarity. But I don't think we should keep it and in general there is no benefit (it wasn't there before and no one missed it and even IF you'd be deploying on Windows you could use the Linux Subshell so really there is no benefit).

Considering `docker stack` or `docker swarm` or in general improving Listmonk:

I have seen that there are flags undocumented but working fine such as --yes and --idempotent. They totally make sense and I added them to the README.md.

Now running a binary twice really is beyond convenient for deployable stacks on docker. It's doable manually for sure but really this isn't how Cloud usually works and makes the process very inefficient.

I noticed that in theory you could run the process twice in one container with using the --idempotent together with --install. But then again: Unnecessary. However also I didn't want to change the default behavior as I find that dangerous in general as it would be a breaking change.

So, what I did:

I introduced a new flag --continue-after-install which is also passed to install.go. It makes sure that listmonk will run install but then not exit but keep-alive. Together with --idempotent this allows for very convenient deployments which I also documented in installation.md and also added a sample for usage with docker secrets as well as providing a downloadable sample docker-compose.yml.

Considering SMTP Settings UX improvement:

It wasn't clear to me that "Testing" currently only works when the password is unsaved. I find this totally okay but it's UX-wise VERY confusing.

So now if you try to test it you'll get this, which is nice:

activenode · 2023-07-28T19:22:53Z

.github/workflows/release.yml

@@ -9,6 +9,9 @@ jobs:
  goreleaser:
    runs-on: ubuntu-latest
    steps:
+      - name: Set up QEMU


This is needed for buildx and goreleaser to build multi-arch. Also documented in https://goreleaser.com/cookbooks/multi-platform-docker-images/

activenode · 2023-07-28T19:23:12Z

.github/workflows/release.yml

@@ -36,6 +39,10 @@ jobs:
        run: |
          make dist

+      - name: Check Docker Version


Doesn't harm, is helpful for debugging failed actions

activenode · 2023-07-28T19:23:33Z

.goreleaser.yml

@@ -1,6 +1,8 @@
 env:
  - GO111MODULE=on
  - CGO_ENABLED=0
+  - GITHUB_ORG=knadh


This allows people to fork and change to their own username and hence test build and publish

activenode · 2023-07-28T19:23:50Z

.goreleaser.yml

-      - freebsd
-      - openbsd
-      - netbsd
+      # - freebsd


Obvious. Feel free to remove but I actually like it for clarity reasons.

activenode · 2023-07-28T19:24:46Z

.goreleaser.yml

@@ -41,123 +44,123 @@ dockers:
    goos: linux
    goarch: amd64
    ids:
-    - listmonk
+      - listmonk


Really not much has changed here. It's all just got better through the variable usage. And because that was properly formatted etc it looks like everything changed but barely anything changed.

activenode · 2023-07-28T19:25:14Z

README.md

@@ -2,7 +2,7 @@

 [![listmonk-logo](https://user-images.githubusercontent.com/547147/231084896-835dba66-2dfe-497c-ba0f-787564c0819e.png)](https://listmonk.app)

-listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a PostgreSQL (⩾ v9.4) database as its data store.
+listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a PostgreSQL (⩾ 12) database as its data store.


This one is kinda funny because I didn't change that version. Do you know why it shows changed?

activenode · 2023-07-28T19:26:20Z

cmd/init.go

 	)

-	// Memory / alloc stats.
-	runtime.ReadMemStats(&mem)
+	si.GetSysInfo() // OS info.


This is the Utsname replacement basically. It doesn't die on multi-arch.

activenode · 2023-07-28T19:26:33Z

cmd/install.go

@@ -17,7 +17,7 @@ import (

 // install runs the first time setup of creating and
 // migrating the database and creating the super user.
-func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempotent bool) {
+func install(lastVer string, db *sqlx.DB, fs stuffbin.FileSystem, prompt, idempotent bool, continueAfterInstall bool) {


just passing the flag here

activenode · 2023-07-28T19:26:46Z

cmd/install.go

+				lo.Println("exiting now (use --continue-after-install to continue)")
+				os.Exit(0)
+			} else {
+				// simply end the function to avoid resetting up the database and pulling queries


obvious, see comment.

activenode · 2023-07-28T19:27:06Z

cmd/main.go

-		os.Exit(0)
+		install(migList[len(migList)-1].version, db, fs, !ko.Bool("yes"), ko.Bool("idempotent"), ko.Bool("continue-after-install"))
+
+		if !ko.Bool("continue-after-install") {


making sure old behaviour is kept when the flag is not set.

activenode · 2023-07-28T19:28:14Z

docker-compose.stack-demo.yml

@@ -0,0 +1,75 @@
+version: "3.7"


Yeah this file is the demo file for what is documented in the docs.

Thunderbottom · 2023-08-03T19:14:54Z

Hi, thanks for the contribution. Will test this and get back to you shortly.

knadh · 2023-08-09T05:50:55Z

Thanks for the PR. However, this contains many unrelated changes. Please send separate PRs for those.

I introduced a new flag --continue-after-install

Unsure if we should introduce yet another flag. Wouldn't ./listmonk --install --yes && ./listmonk work? @mr-karan

mr-karan · 2023-08-09T06:27:28Z

Wouldn't ./listmonk --install --yes && ./listmonk work

+1

activenode · 2023-08-09T06:55:20Z

I can surely split the PR but the command you are proposing will truncate the DB completely when docker service restarts. And Docker Service Restarts are amongst the most common thing on earth :)

So what you are saying is "run 2 commands and you're fine".

If you wanted what you proposed it would look like this:

./listmonk --install --yes  --idempotent && ./listmonk

Now when you need to add config, which is the normal use case when using docker secrets etc. this becomes easily a large and cumbersome command where you are workarounding running "one process" (that is what docker does -> running exactly 1 process) needs to be worked around with something like sh -c and this again brings downsides because you are starting a new shell and now need to forward the output.

Running 2 processes in CMD is also not a good practice. Sure, you could fake it being "one process" by putting it to a bash script but then the question is: isn't this overcomplicating what doesn't need to be overcomplicated?

I really really recommend having this in one single process, there's not much need to split those apart. ORM's behave the same way: You startup the single process and it will setup the structure if needed. That is best practice.

activenode · 2023-08-09T07:02:27Z

The only other thing I can propose as compromise is making --idempotent the default to true as well as --install and then make my new flag essentially the default behaviour.

In that case we would get all what we want: Less flags, one process, installs IF not installed yet.

What do you prefer?

edit:
Last but not least another reason why 2 commands are bad: In Docker the default use-case when having orchestrators such as K8S or Swarm is that you create replicas. With the sample that you provided we would run n times replicas with --install first, truncating the db whilst another one is writing.

edit 2:
personally I would prefer the option to have --idempotent + --install + --run-after-install being the default but that would kinda be a breaking change so from an architectural view I'd stick with the newly introduced flag.

Just tell me what of those you want but please don't tell me that running 2 processes is a good way to go because I can tell you: In orchestrated Docker Environments it is definitely not :)

knadh · 2023-08-09T07:42:54Z

Ah, we cannot change the meaning of --yes. That breaks compatibility.

./listmonk --install --yes --idempotent && ./listmonk is a long command, but it's composition at work. Adding a new --install-and-continue is adding a hardcoded mode into listmonk which ignores external composability.

I think a slightly long and verbose command is a fair trade-off here.

activenode · 2023-08-09T07:49:28Z

Can you elaborate what you mean with breaking external composability?

Because for orchestrated Docker running 2 commands effectively means that you have to use sh -c which is a workaround and not good practice and I am trying to make Listmonk usable for the standard orchestrated use-case without introducing breaking changes.

Or do you have any better proposal? Like I really want the docker CMD to be running the exact process and not 2.

Where do you see the benefit in installing first and only installing (besides talking about breaking changes now)?
I am having a really hard time understanding why someone would benefit from that.

If we introduce a new major version update I don't see a problem with having breaking changes.
The existing default that --install truncates the DB is actually quite dangerous IMO. I vanished the whole DB one time with that. Luckily with not much data.

Also I would like to understand what you mean with hardcoded mode . I mean at the end of the day all of those flags are "hardcoded" in that sense?

activenode · 2023-08-09T08:51:59Z

Either way, first things first I will split the PR to make releases work again

activenode · 2023-08-13T07:45:23Z

@knadh @Thunderbottom @mr-karan I cleaned up this PR to be only the multi-arch fix. I will open others for the other things.

activenode · 2023-08-13T07:46:24Z

However I rename it now because it really isn't real swarm support as I understand it simply because we don't have the aforementioned and discussed --continue-after-install flag.

knadh · 2023-08-13T07:52:55Z

Hi @activenode. Apologies, there are so many commits and comments now, I'm confused! Please rebase with the latest master and amend the PR with fewer/squashed commits.

activenode changed the title ~~Docker Multi Arch Builds and Swarm Support~~ WIP: Docker Multi Arch Builds and Swarm Support Jul 28, 2023

activenode mentioned this pull request Jul 28, 2023

feat: add multiarch docker build support #1344

Merged

activenode commented Jul 28, 2023

View reviewed changes

activenode changed the title ~~WIP: Docker Multi Arch Builds and Swarm Support~~ Docker Multi Arch Builds and Swarm Support Jul 28, 2023

activenode changed the title ~~Docker Multi Arch Builds and Swarm Support~~ Docker Multi Arch Builds Fix Aug 13, 2023

activenode force-pushed the feature/docker-multi-arch-and-swarm-flags-pr branch from 7d3c226 to ffaba95 Compare August 13, 2023 07:57

activenode closed this Aug 13, 2023

activenode force-pushed the feature/docker-multi-arch-and-swarm-flags-pr branch from ffaba95 to f3e9b08 Compare August 13, 2023 07:59

activenode mentioned this pull request Aug 13, 2023

Docker Multi Arch (esp. ARM) builds: Improving Build File #1451

Merged

Docker Multi Arch Builds Fix #1414

Docker Multi Arch Builds Fix #1414

Uh oh!

Conversation

activenode commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Considering Multi-Arch:

QEMU missing

Multi-Arch Problems 1

Multi-Arch Problems 2

Considering docker stack or docker swarm or in general improving Listmonk:

So, what I did:

Considering SMTP Settings UX improvement:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Thunderbottom commented Aug 3, 2023

Uh oh!

knadh commented Aug 9, 2023

Uh oh!

mr-karan commented Aug 9, 2023

Uh oh!

activenode commented Aug 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

activenode commented Aug 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

knadh commented Aug 9, 2023

Uh oh!

activenode commented Aug 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

activenode commented Aug 9, 2023

Uh oh!

activenode commented Aug 13, 2023

Uh oh!

activenode commented Aug 13, 2023

Uh oh!

knadh commented Aug 13, 2023

Uh oh!

Uh oh!

activenode commented Jul 28, 2023 •

edited

Loading

Considering `docker stack` or `docker swarm` or in general improving Listmonk:

activenode commented Aug 9, 2023 •

edited

Loading

activenode commented Aug 9, 2023 •

edited

Loading

activenode commented Aug 9, 2023 •

edited

Loading