-
|
I have a workflow building several docker images and only publishing those images if all builds succeed. The obvious way to solve this was to have a build job with a matrix of the image names and then a publish job with the same matrix that “needs” the build job. This works fine. However now I also want to generate outputs during build for each image that I want to re-use during the publish job for the same image. I know that I can work around this using artefacts but if these weren’t matrix jobs I could just do it using outputs. Is there a special syntax to do access outputs from a matrix job or are matrix jobs simply incapabl of generating outputs? I can’t find any explanation of this in the docs and it doesn’t seem like the outputs could be named dynamically since they need to be explicitly defined at the job level. |
Beta Was this translation helpful? Give feedback.
Replies: 11 comments 17 replies
-
|
Hey Alan, matrix jobs can set outputs but they will overwrite each other. The last one wins. As a workaround, you can add a unique parameter for each matrix job and use that to set outputs with different names: [BUG] Jobs output should return a list for a matrix job - #15 by Simran-B |
Beta Was this translation helpful? Give feedback.
-
|
That link is broken? |
Beta Was this translation helpful? Give feedback.
-
facing this problem now. @Simran-B can you share workaround again? |
Beta Was this translation helpful? Give feedback.
-
|
I'm unsure about this but the original link in @Simran-B's post might've referred to the following discussion: Jobs need a way to reference all outputs of matrix jobs #17245 |
Beta Was this translation helpful? Give feedback.
-
|
Hey thanks Simran, looks like I’ll have to duplicate the matrix list for the outputs then. That solution seems obvious in hindsight but didn’t occur to me. |
Beta Was this translation helpful? Give feedback.
-
|
Simran-B:
This is not great. I hope the GitHub team will fix this. |
Beta Was this translation helpful? Give feedback.
-
|
Been trying some workarounds for days now, and no it is no possible. The only think I can think of is to set a secret to a repository and check the value afterward. So you can't use job level env, you can't set output dynamicaly, thoses values are overwritten by each matrix jobs... |
Beta Was this translation helpful? Give feedback.
-
|
I'm considering storing data in the git repo directly. This post from Simon Willison sparked this idea: https://simonwillison.net/2020/Oct/9/git-scraping/ |
Beta Was this translation helpful? Give feedback.
-
|
One of the options is to pass outputs via output artifacts, using GH actions upload-artifact and download-artifact: name: "CI/CD"
on:
push:
branches: [master]
defaults:
run:
shell: bash
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
kv:
- key: one
val: ONE
- key: two
val: TWO
- key: three
val: THREE
steps:
- id: one
run: |
mkdir -p pipelines
echo ${{ matrix.kv.val }} >> outputs/${{ matrix.kv.key }}.txt
- uses: actions/upload-artifact@v3
with:
name: outputs
path: outputs/*.txt
getval:
runs-on: ubuntu-latest
needs: [test]
steps:
- uses: actions/checkout@v3
- name: Load outputs
uses: actions/download-artifact@v3
with:
name: outputs
path: outputs
- run: |
ls outputs
cat outputs/*.txt |
Beta Was this translation helpful? Give feedback.
-
|
Right, but before someone will become a bit too enthusiastic by this comment here, it is also good to include follow-up to that quote, from the same source:
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, good point: you need to make sure to upload under different names. |
Beta Was this translation helpful? Give feedback.
-
|
This no longer works with the latest version of the upload-artifact unfortunately.
|
Beta Was this translation helpful? Give feedback.
-
|
@p5 I'm literally staring at this exact line of documentation today and trying to find a solution. I wish that we could bring all the filesystems systems from each matrix branch back down into a single filesystem. |
Beta Was this translation helpful? Give feedback.
-
|
I found a workaround without any corruption risk as I write a different artifact for each matrix job. |
Beta Was this translation helpful? Give feedback.
-
|
That works because name: Works
on:
push:
branches: ["**"]
jobs:
parallel:
runs-on: ubuntu-latest
strategy:
fail-fast: true
matrix:
option:
- a
- b
steps:
- name: output
id: output
run: |
echo "${{ matrix.option }}=${{ matrix.option }}" >> $GITHUB_OUTPUT
outputs:
a: ${{ steps.output.outputs.a }}
b: ${{ steps.output.outputs.b }}
finally:
needs: parallel
runs-on: ubuntu-latest
strategy:
matrix:
option: ${{ needs.parallel.outputs.* }}
steps:
- name: checkout
uses: actions/checkout@v3
- name: debug
run: echo '${{ toJSON(needs.parallel.outputs) }}' |
Beta Was this translation helpful? Give feedback.
-
|
neat, but this solution has a limitation that the output keys must be defined statically in yaml, so for variable set of matrix jobs it won't work :( |
Beta Was this translation helpful? Give feedback.
-
|
Facing this problem too. My workflow is dynamic matrix jobs so that's not possible to pre-define outputs statically |
Beta Was this translation helpful? Give feedback.
-
|
I have used this action to solve the problem, works well with dynamic matrix jobs: https://github.com/marketplace/actions/matrix-outputs-write |
Beta Was this translation helpful? Give feedback.
-
|
We've made a couple of actions to deal with this issue: |
Beta Was this translation helpful? Give feedback.
-
|
@andy-maier it should be fixed with version https://github.com/hoverkraft-tech/ci-github-common/releases/tag/0.8.1 Please let me know |
Beta Was this translation helpful? Give feedback.
-
|
How is uniqueness guaranteed with that action? I assume it will keep it within github worklow scope? So if someone else starts same workflow, it would not clash? Let say I have matrix of |
Beta Was this translation helpful? Give feedback.
-
|
@oerp-odoo The GitHub Actions workflow ensures artifact uniqueness by concatenating the workflow's run_id, run_number, and a user-provided artifact-name. Users can enhance this uniqueness by supplying a distinct artifact-name for each run, further ensuring that each artifact is uniquely identified across all runs and workflows. This strategy effectively prevents any clashes or overwrites, maintaining the integrity and traceability of artifacts, especially when multiple workflows are executed concurrently or under similar configurations. https://github.com/hoverkraft-tech/ci-github-common/blob/main/actions/set-matrix-output/README.md (doc updated) |
Beta Was this translation helpful? Give feedback.
-
|
@neilime thanks. One more question. How long those artifacts persist? I assume if workflow ends, artifacts are gone? I would not want them to persist longer than it needs to. And I probably can't rely on delete after fetching artifact option, because I might need to fetch it multiple times. |
Beta Was this translation helpful? Give feedback.
-
|
If you are using the action https://github.com/hoverkraft-tech/ci-github-common/tree/main/actions/get-matrix-outputs There is an input to delete or not he artifact after reading If you want to delete the artifact on a specific case you can set the remove-artifact input to false and then remove it later in your workflow when you want (using geekyeggo/delete-artifact action by example) - id: create-artifact
uses: hoverkraft-tech/ci-github-common/actions/set-matrix-output@v0.7.5
with:
value: "test value"
artifact-name: "test"
- uses: hoverkraft-tech/ci-github-common/actions/get-matrix-outputs@v0.7.5
with:
artifact-name: "test"
remove-artifact: false
# Delete artifact
- uses: geekyeggo/delete-artifact@v2
with:
name: ${{ steps.create-artifact.outputs.artifact-name }}
|
Beta Was this translation helpful? Give feedback.
-
|
I've developed an action to workaround this problem. It only requires adding one additional step to your job matrix and setting and output: https://github.com/beacon-biosignals/matrix-output |
Beta Was this translation helpful? Give feedback.
-
|
My workaround is to upload a unique artifact for each matrix job and then I download all artifacts with a |
Beta Was this translation helpful? Give feedback.
Hey Alan, matrix jobs can set outputs but they will overwrite each other. The last one wins. As a workaround, you can add a unique parameter for each matrix job and use that to set outputs with different names: [BUG] Jobs output should return a list for a matrix job - #15 by Simran-B