Add `execution.test.fail` to fail a test, and let it finish #4672

oleiade · 2025-04-04T13:35:25Z

What?

This Pull Request addresses #4062.

Namely, it adds the execution.test.fail method, which marks a test as failed, and let it finish its execution nonetheless, and eventually exit with a specific exit code (110).

The following script:

import http from "k6/http";
import exec from "k6/execution";

export const options = {
	iterations: 10,
};

export default function () {
	http.get("https://quickpizza.grafana.com");

	if (exec.vu.iterationInInstance === 3) {
		console.log("Marking the test as failed from iteration: 3");
		exec.test.fail(
			`iteration ${exec.vu.iterationInInstance}: marked the test as failed`,
		);
	}

	console.log(
		`iteration ${exec.vu.iterationInInstance}: print after (maybe) fail`,
	);
}

Ran with the following command:

go build go.k6.io/k6 && ./k6 run script.js; echo $status

Leads to the following output:


         /\      Grafana   /‾‾/
    /\  /  \     |\  __   /  /
   /  \/    \    | |/ /  /   ‾‾\
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/

     execution: local
        script: script.js
        output: -

     scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
              * default: 10 iterations shared among 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)

INFO[0000] iteration 0: print after (maybe) fail         source=console
INFO[0000] iteration 1: print after (maybe) fail         source=console
INFO[0000] iteration 2: print after (maybe) fail         source=console
INFO[0000] Marking the test as failed from iteration: 3  source=console
ERRO[0000] test marked as failed: iteration 3: marked the test as failed
INFO[0000] iteration 3: print after (maybe) fail         source=console
INFO[0001] iteration 4: print after (maybe) fail         source=console
INFO[0001] iteration 5: print after (maybe) fail         source=console
INFO[0001] iteration 6: print after (maybe) fail         source=console
INFO[0001] iteration 7: print after (maybe) fail         source=console
INFO[0001] iteration 8: print after (maybe) fail         source=console
INFO[0001] iteration 9: print after (maybe) fail         source=console


  █ TOTAL RESULTS

    HTTP
    http_req_duration.......................................................: avg=129.22ms min=127.9ms  med=129.17ms max=130.83ms p(90)=130.51ms p(95)=130.67ms
      { expected_response:true }............................................: avg=129.22ms min=127.9ms  med=129.17ms max=130.83ms p(90)=130.51ms p(95)=130.67ms
    http_req_failed.........................................................: 0.00%  0 out of 10
    http_reqs...............................................................: 10     5.763858/s

    EXECUTION
    iteration_duration......................................................: avg=173.46ms min=128.07ms med=129.43ms max=569.82ms p(90)=174.89ms p(95)=372.36ms
    iterations..............................................................: 10     5.763858/s
    vus.....................................................................: 1      min=1       max=1
    vus_max.................................................................: 1      min=1       max=1

    NETWORK
    data_received...........................................................: 32 kB  19 kB/s
    data_sent...............................................................: 1.0 kB 601 B/s




running (00m01.7s), 0/1 VUs, 10 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  00m01.7s/10m0s  10/10 shared iters
ERRO[0001] test run was marked as failed
110

Note that the test is marked as failed on iteration 3, but doesn't keep neither the 3rd iteration, nor the following ones to be interupted. However, the fail method immediately logs that the test was marked as failed, and eventually, the test run prints an error reporting the test as marked as failed, and exits with code 110.

Why?

The goal of #4062 and this PR specifically is to cater to functional testing needs, and specifically the new assertions API, and opens the door to soft assertions: assertions that do not interrupt the execution, but can be detected and treated as a failed test nonetheless.

Checklist

I have performed a self-review of my code.
I have commented on my code, particularly in hard-to-understand areas.
I have added tests for my changes.
I have run linter and tests locally (make check) and all pass.

Checklist: Documentation (only for k6 maintainers and if relevant)

Please do not merge this PR until the following items are filled out.

I have added the correct milestone and labels to the PR.
I have updated the release notes: link
Add execution.test.fail k6-docs#1941
I have updated or added an issue to the TypeScript definitions: grafana/k6-DefinitelyTyped#NUMBER if applicable

Related PR(s)/Issue(s)

ref #4062

inancgumus

LGTM 👍 Will you add t.Parallel to the subtests? Is there a reason for skipping parallel tests?

ankur22

LGTM 🚀

joanlopez · 2025-04-08T14:22:58Z

lib/test_state.go

@@ -26,6 +27,44 @@ type TestPreInitState struct {
 	TracerProvider *trace.TracerProvider
 	Usage          *usage.Usage
 	SecretsManager *secretsource.Manager
+
+	// FIXME (@oleiade): is this the way?


I'm not 100% sure either, but perhaps this should belong to Runner or TestRunState, as TestPreInitState sounds more the place for things that remain static right after initialization 🤔

Let me make another pass at it, initially I stored it in the TestPreInitState because that's directly accesible at the VU level, but it might be better indeed to bury it a bit more 👀

joanlopez · 2025-04-08T14:35:10Z

LGTM 👍 Will you add t.Parallel to the subtests? Is there a reason for skipping parallel tests?

Yeah, I agree that duplicating the test setup to permit the parallel execution of tests would be nice, but either way doesn't sound such a big deal.

joanlopez

Despite the two comments I left, it looks mostly good and ready to be shipped!

And, to be clear, I'm not 100% sure neither about what's the best place for TestStatus.

oleiade · 2025-04-09T07:38:34Z

Cheers folks, I'll just make a quick pass on the code to see if I can avoid using the TestPreInitState and I'll refactor the test to be in Parallel 👍🏻

oleiade · 2025-04-10T09:54:58Z

Heads-up @inancgumus @ankur22 @joanlopez 👋🏻 :

I have updated the tests according to your comments folks, and they are now parallel
I have spent a couple hours looking into moving the TestStatus down into the Runner/TestRunState, and it would most likely need a pretty big refactor to be possible, and I don't think it's worth the effort or the risk as we're looking to release v1. Thus, and considering I have the plan to make a refactor pass on the run command during cooldown/pre-release, I'll open an issue to look further into it, and would merge the PR as-is 👍🏻

More Context: Like brought up by Joan, a better place could potentially be to have the TestStatus pointer stored in the TestRunState or even the Runner.

For the feature to behave the way we want, we need to ensure that the k6 process only has a single instance of the TestStatus throughout the whole execution, and that we pass that pointer down to the VU state, where the execution module may or may not set it to fail based on the user-script.

However, down the road, the element that sets the State for VUs, is the Runner (which, is actually a VUFactory, rather an actual runner that would run code, and if I remember correctly for multiple executors, there would be multiple runners), and the only piece of state the runner knows about currently is... the PreInitState.

In a future refactor/iteration, we would need to look into whether we want to change that 🙇🏻

ankur22

LGTM 🚀

joanlopez · 2025-04-10T13:13:19Z

However, down the road, the element that sets the State for VUs, is the Runner (which, is actually a VUFactory, rather an actual runner that would run code, and if I remember correctly for multiple executors, there would be multiple runners), and the only piece of state the runner knows about currently is... the PreInitState.

That's what worried me, thus why I suggested exploring that avenue, but with caution. Thanks, and apologies! 🙏🏻

inancgumus · 2025-04-10T13:59:47Z

2. I have the plan to make a refactor pass on the run command during cooldown/pre-release, I'll open an issue to look further into it, and would merge the PR as-is 👍🏻

Thanks for the explanation, @oleiade 🙇 Helpful! Is it worth opening that issue now so we remember?

oleiade · 2025-04-10T14:08:39Z

Thanks for holding me accountable to it @inancgumus 🙇🏻

I just opened a dedicated issue: #4681

oleiade added 4 commits April 4, 2025 15:33

Add TestStatus construct to track the fail/pass state of a test

bd6a571

Add TestStatus to the application's state

dedf33b

Add execution.test.fail() method

7669c63

Ensure the run command catches failed tests and returns exit code 110

fcd9dd9

oleiade added the enhancement label Apr 4, 2025

oleiade added this to the v1.0.0 milestone Apr 4, 2025

oleiade self-assigned this Apr 4, 2025

oleiade requested a review from a team as a code owner April 4, 2025 13:35

oleiade requested review from inancgumus and joanlopez and removed request for a team April 4, 2025 13:35

inancgumus previously approved these changes Apr 4, 2025

View reviewed changes

Deactivate parallel tests for execution.test.fail

97d4c66

oleiade dismissed inancgumus’s stale review via 97d4c66 April 4, 2025 14:03

Deactivate parallel subtests for execution.test.fail

8dff70c

ankur22 previously approved these changes Apr 7, 2025

View reviewed changes

inancgumus previously approved these changes Apr 7, 2025

View reviewed changes

joanlopez reviewed Apr 8, 2025

View reviewed changes

joanlopez previously approved these changes Apr 8, 2025

View reviewed changes

Apply Pull Request suggestions

19ff1ab

oleiade dismissed stale reviews from joanlopez, inancgumus, and ankur22 via 19ff1ab April 10, 2025 09:44

Merge branch 'master' into mark-test-as-failed

f55b025

Fix linter

4184a06

oleiade requested review from inancgumus and joanlopez April 10, 2025 09:56

oleiade requested a review from ankur22 April 10, 2025 09:56

ankur22 approved these changes Apr 10, 2025

View reviewed changes

joanlopez approved these changes Apr 10, 2025

View reviewed changes

oleiade merged commit 80fa0df into master Apr 10, 2025
28 checks passed

oleiade deleted the mark-test-as-failed branch April 10, 2025 13:17

oleiade mentioned this pull request Apr 10, 2025

Refactor the TestStatus out of PreInitState #4681

Open

oleiade mentioned this pull request Apr 10, 2025

Add the ability to mark a test as failed, and let it finish its execution with a specific non-zero exit code #4062

Closed

inancgumus mentioned this pull request Apr 29, 2025

Add execution.test.fail grafana/k6-docs#1941

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `execution.test.fail` to fail a test, and let it finish #4672

Add `execution.test.fail` to fail a test, and let it finish #4672

Uh oh!

oleiade commented Apr 4, 2025 •

edited by inancgumus

Loading

Uh oh!

inancgumus left a comment •

edited

Loading

Uh oh!

ankur22 left a comment

Uh oh!

joanlopez Apr 8, 2025

Uh oh!

oleiade Apr 9, 2025

Uh oh!

joanlopez commented Apr 8, 2025

Uh oh!

joanlopez left a comment

Uh oh!

oleiade commented Apr 9, 2025

Uh oh!

oleiade commented Apr 10, 2025 •

edited

Loading

Uh oh!

ankur22 left a comment

Uh oh!

joanlopez commented Apr 10, 2025

Uh oh!

Uh oh!

inancgumus commented Apr 10, 2025 •

edited

Loading

Uh oh!

oleiade commented Apr 10, 2025

Uh oh!

Uh oh!

Add execution.test.fail to fail a test, and let it finish #4672

Add execution.test.fail to fail a test, and let it finish #4672

Uh oh!

Conversation

oleiade commented Apr 4, 2025 • edited by inancgumus Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

Checklist

Checklist: Documentation (only for k6 maintainers and if relevant)

Related PR(s)/Issue(s)

Uh oh!

inancgumus left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankur22 left a comment

Choose a reason for hiding this comment

Uh oh!

joanlopez Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

oleiade Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

joanlopez commented Apr 8, 2025

Uh oh!

joanlopez left a comment

Choose a reason for hiding this comment

Uh oh!

oleiade commented Apr 9, 2025

Uh oh!

oleiade commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ankur22 left a comment

Choose a reason for hiding this comment

Uh oh!

joanlopez commented Apr 10, 2025

Uh oh!

Uh oh!

inancgumus commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oleiade commented Apr 10, 2025

Uh oh!

Uh oh!

Add `execution.test.fail` to fail a test, and let it finish #4672

Add `execution.test.fail` to fail a test, and let it finish #4672

oleiade commented Apr 4, 2025 •

edited by inancgumus

Loading

inancgumus left a comment •

edited

Loading

oleiade commented Apr 10, 2025 •

edited

Loading

inancgumus commented Apr 10, 2025 •

edited

Loading