Skip to content

GitHub Actions tests workflow improvments - input #8787

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

duttonw
Copy link
Contributor

@duttonw duttonw commented Apr 11, 2025

Hi @amercader ,

Here's some info you may want to take as a whole or cherry-pick parts from.

I'm happy for you to tweak/edit any of this (like the solr image on ghcr.io that is hash based for cicd testing).
Be aware that ghcr.io is org/user based and this is designed to be fork compatible (which is the reason to have dangling tag removal which is still wip since you can slowly manually cleanup via gui or via others gist scripts)

Fixes #8783

Proposed fixes:

  • Build solr container so schema changes can be tested atomically these are stored in GitHub ghcr.io package system
    • only update if schema (hash) and solr version are different
  • fix rollup of reports
  • we build docs, keep them for a day for review
  • extracted most bash scripts out of yaml
  • reintroduced pip cache for speedup
  • move most pytest options into toml file
  • retrys now 0, not 1 so flaky tests are more visible.
  • .test_durations.gz updated with latest master run

Features:

  • includes tests covering changes
  • includes updated documentation
  • includes user-visible changes
  • includes API changes
  • includes bugfix for possible backport

Please [X] all the boxes above that apply

ACTOR_TYPE=$(curl -s -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
https://api.github.com/users/${GITHUB_ACTOR} | jq -r .type)
if [[ "$ACTOR_TYPE" == "Organization" ]]; then
ACTOR_API_PATH=orgs/${GITHUB_ACTOR}
Copy link
Contributor Author

@duttonw duttonw Apr 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security wise, PR's cross org are not allowed to deploy to ghcr.io package system unless that user is a 'member' of said org.

This could be improved to use local org ckan if the tag matches (usually it would) but if there is schema changes we won't have a tag and would need to reference the container made by the incoming user if possible. (there is some risk on this that its been 'poisoned' by a bad actor but unsure what it could do if anything)

Since schema changes are so rare we would be referencing the exact tagged version from repo owner majority of the time.

(unsure if a pr rasied by an actor would be encoded into the github_token to allow a push { it could but needs further testing} )

This is now working for PR from user to org and should also do org to org.

It will need more testing on creating containers inside the ckan org (as well as possible settings if not enabled in the org/repo level)

Its also a pita that they made a differentiation between user and orgs for package api.

This could could be moved into a shell script and have ENV_ARGS passed through but I've spent enough time today on this and need to give it a rest.


I also investigated having the docker container locally built and not use a container manager but due to running the build in a container image (not root gh) it would not be possible to add it to auto dns resolution and would then need some hackery with /etc/hosts and local port forwarding to 'emulate' it being on ckan-solr

@amercader
Copy link
Member

Many thanks @duttonw , I've pulled some stuff into #8783.

In the future please split separate changes in separate commits as it makes it easier to cherry-pick just what's needed.

@amercader amercader closed this Apr 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants