/kind feature
/area testing

Recently, we switched to GitHub Actions for our CI/CD pipelines, thanks a lot again @tenzen-y for driving this.

Since we have limitations now: 20 concurrent jobs and we haven't set AWS EC2 instances for our workers yet, we need to do some improvements to reduce execution time.

I think, we can try to do the following:

1. Should we run postgres test only for Random search experiment ? We run 3 Trials for Random experiment, so we can verify that DB works properly.

2. Can we build only the required suggestions images for each e2e test ? As I can see, build step takes around 15 min which is more than half of e2e.

3. @tenzen-y Are there any specific requirements why we clean cache for our build image after each e2e run ?

4. Do we need to build images for linux/amd64 if that is verified as part of e2e ?

5. In the longterm/separate tracking issue we can also do this:

   • Run only required Experiments when appropriate source code has been changed (what we've done with Katib UI).
   • Run all experiments test in periodic manner, e.g. once a day. For Pull Request test we can use only few e2e experiments.
   • Use Katib SDK instead of this script to run e2e, similar to the Training Operator. So we can verify that our SDK is working.

@kubeflow/wg-training-leads @tenzen-y @anencore94 Are there any other improvements that you have in your mind ?

GitHub Actions improvements checklist

I can identify the following improvements:

• Run postgres e2e only for random search.
• Use Katib SDK to create E2E script.
• Disable workflow when the new commit is published by using cancel-in-progress API.
• Use Docker cache when building our images.
• Remove linux/amd64 build from the pre-commit check since we verify this in E2E test.
• Identify experiments for E2E from the appropriate code changes.
• Run all E2E test only on the pre-releases or in the periodic manner.

Please let me know if we should add more items @johnugeorge @anencore94 @terrytangyuan @tenzen-y @gaocegege

Love this feature? Give it a 👍 We prioritize the features with the most 👍