Skip to content

Conversation

smoser
Copy link
Contributor

@smoser smoser commented Sep 6, 2023

This is an attempt at fixing nightly c-i similar to #496
but with more re-use and building more nightkly.

@hallyn
Copy link
Contributor

hallyn commented Sep 6, 2023

Seems reasonable.

@smoser
Copy link
Contributor Author

smoser commented Sep 6, 2023

So it looks like this doesn't work as intended.

I made the nightly workflow run on PR in the top most commit.

Both of those show in their test output:

ok 19 all container contents must be accounted for # skip test_all_container_contents_must_be_accounted_for is slow. Set SLOW_TEST=true to run.
ok 20 bom tool should work inside run # skip test_bom_tool_should_work_inside_run is slow. Set SLOW_TEST=true to run.

Further investigation shows that I had SLOW_TESTS (plural) in the workflow 'env' while the code is looking at SLOW_TEST (singular).

Change made, pushed, lets see.

After this runs i have to still drop the top-most commit so that nightly doesn't run on PR.

@smoser smoser force-pushed the fix/nightly-ci branch 4 times, most recently from 1ffe228 to 7b76be8 Compare September 6, 2023 18:01
run: |
make check
build:
uses: ./.github/workflows/build.yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is "slow-test" actually set to true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is "slow-test" actually set to true?

It defaults to true. (.github/workflows/build.yaml).

it seems that "run all the tests" is a better default. That way in a release and in a nightly we get all the tests run. the only place we don't get them all run is in the 'ci.yaml' caller.

The build workflow was being 'used' by callers in ci and release.
Here also make 'nightly' a caller.

Additionally,
 * add 'slow-test' input to build.yaml, defaulting to true.
   Set that to false in ci.yaml so we do not run the slow tests
   on all PRs.
   slow-test value gets put into environment variable SLOW_TEST
   which is read by the test harness.
 * make build-id a scalar, not an array.
 * move build-id out of matrix (it is a constant) and reference
   it as an input instead.
 * Provide default values for 'privilege-level' and 'go-version'
   in build.yaml so the callers do not have to specify those.

Note that we still *do* specify the privilege-level and go-version
in a release build.  This is, I believe, because we want a consistent
thing that gets restored in the cache.

It is unfortunate, but as a result, we aren't testing with unpriv
during a release build.

Signed-off-by: Scott Moser <smoser@brickies.net>
@smoser smoser force-pushed the fix/nightly-ci branch 2 times, most recently from f69a108 to 7a66c8a Compare September 6, 2023 19:59
@smoser
Copy link
Contributor Author

smoser commented Sep 6, 2023

both of the slow test runs failed. see here.

  • unpriv failed:

    not ok 48 elasticsearch in 177sec
    # (from function `stacker' in file test/helpers.bash, line 87,
    #  in test file test/convert.bats, line 71)
    #   `stacker build -f stacker.yaml --substitute-file stacker-subs.yaml --substitute IMAGE=elasticsearch' failed
    
  • priv failed:

    not ok 7 mount + umount works in 49sec
    # (from function `stacker' in file test/helpers.bash, line 87,
    #  from function `basic_test' in file test/atomfs.bats, line 28,
    #  in test file test/atomfs.bats, line 36)
    #   `basic_test' failed
    # Debug mode: 
    

maybe the 'mount + umount' failure is unrelated as it normally passes, but the other failure is in a test marked slow.

@smoser
Copy link
Contributor Author

smoser commented Sep 7, 2023

I'd like to ask that this be landed. nightly c-i will still be broken, but this PR gets us closer.
Once landed, I think you'll be able to run the nightly manually on a branch to debug/fix it.
See https://docs.github.com/en/actions/using-workflows/manually-running-a-workflow for how to do that.

@hallyn hallyn merged commit 0fe7ea8 into project-stacker:main Sep 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants