I currently don't have the mental capacity, so a full review will need to wait until tomorrow, but:

uses the aliases stable and oldstable

Does this work with the caching mechanism?

# https://github.com/marketplace/actions/cache
- name: Cache Go modules
  uses: actions/cache@v3
  with:
    # ...
    key: ${{ runner.os }}-${{ matrix.go-version }}-go-${{ hashFiles('**/go.sum') }}
    restore-keys: |
      ${{ runner.os }}-${{ matrix.go-version }}-go-

Now with matrix.go-version being stable, the cache key will only change if the go.sum files change. This would only be the case for dependency updates, since the minimum Go version is only present in the go.mod file...

The modules don't depend on the Go version, so the cache can stay the same between two different versions of Go.

But in fact since actions/setup-go@v4 the caching is handled by the action, so I will remove the caching step.
https://github.com/marketplace/actions/setup-go-environment