BrowserContext can't be created for reused VUs #4240
Description
Brief summary
If browser-level test starts after no-browser (protocol-level) tests, an error in browser.newContext() may occur.
It happens when browser-level test reuses VU, that was already used for protocol-level test (or simply test without BrowserContext creation).
Use case: hybrid load testing, many protocol-level tests and some browser-level ones (for testing ASPX pages with multiple internal requests for example), https://k6.io/docs/using-k6-browser/running-browser-tests/#run-both-browser-level-and-protocol-level-tests-in-a-single-script
k6 version
k6 v0.47.0 (commit/5ceb210056, go1.21.2, windows/amd64)
OS
Windows 10
Docker version and image (if applicable)
No response
Steps to reproduce the problem
Just run following script:
import { sleep } from 'k6';
import { browser } from 'k6/experimental/browser';
import exec from 'k6/execution';
export const options = {
scenarios: {
protocolLevel: {
executor: 'constant-vus',
exec: 'protocolLevelTest',
vus: 2,
duration: '1s',
},
browserLevel: {
options: {
browser: {
type: 'chromium',
},
},
executor: 'ramping-vus',
exec: 'browserLevelTest',
startVUs: 1,
stages: [
{ duration: '2s', target: 2 },
{ duration: '2s', target: 2 },
],
startTime: '30s',
},
},
};
export function protocolLevelTest() {
console.log(`${getVuInfo()}: protocol-level test dummy`);
sleep(0.4);
}
export async function browserLevelTest() {
try {
browser.newContext();
console.log(`${getVuInfo()}: BrowserContext created`);
} catch (error) {
console.error(`${getVuInfo()}: ${error}`);
}
sleep(1);
}
function getVuInfo() {
return JSON.stringify(exec.vu);
}Expected behaviour
All iterations of all tests have to pass without errors.
Actual behaviour
It seems it takes 30 seconds to free VUs used by scenario with constant-vus executor.
Before that moment browser-level test ramping completely new VUs with no problems for browser.newContext(). But at 31 second browser-level test takes VU that was used for protocol-level test. In each iteration of this reused VU occurs an error when trying to create BrowserContext (see logs below):
browser not found in registry. make sure to set browser type option in scenario definition in order to use the browser module
Output:
execution: local
script: browserVuErrorTest.js
output: -
scenarios: (100.00%) 2 scenarios, 3 max VUs, 1m4s max duration (incl. graceful stop):
* protocolLevel: 2 looping VUs for 1s (exec: protocolLevelTest, gracefulStop: 30s)
* browserLevel: Up to 2 looping VUs for 4s over 2 stages (gracefulRampDown: 30s, exec: browserLevelTest, startTime: 30s, gracefulStop: 30s)
INFO[0000] {"idInInstance":2,"idInTest":2,"iterationInInstance":0,"iterationInScenario":0,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy source=console
INFO[0000] {"idInInstance":1,"idInTest":1,"iterationInInstance":0,"iterationInScenario":0,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy source=console
INFO[0000] {"idInInstance":1,"idInTest":1,"iterationInInstance":1,"iterationInScenario":1,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy source=console
INFO[0000] {"idInInstance":2,"idInTest":2,"iterationInInstance":1,"iterationInScenario":1,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy source=console
INFO[0001] {"idInInstance":2,"idInTest":2,"iterationInInstance":2,"iterationInScenario":2,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy source=console
INFO[0001] {"idInInstance":1,"idInTest":1,"iterationInInstance":2,"iterationInScenario":2,"tags":{"group":"","scenario":"protocolLevel"},"metrics":{"tags":{"group":"","scenario":"protocolLevel"},"metadata":{}}}: protocol-level test dummy source=console
INFO[0030] {"idInInstance":3,"idInTest":3,"iterationInInstance":0,"iterationInScenario":0,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: BrowserContext created source=console
INFO[0031] {"idInInstance":3,"idInTest":3,"iterationInInstance":1,"iterationInScenario":1,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: BrowserContext created source=console
ERRO[0032] {"idInInstance":1,"idInTest":1,"iterationInInstance":3,"iterationInScenario":0,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: GoError: browser not found in registry. make sure to set browser type option in scenario definition in order to use the browser module source=console
INFO[0032] {"idInInstance":3,"idInTest":3,"iterationInInstance":2,"iterationInScenario":2,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: BrowserContext created source=console
ERRO[0033] {"idInInstance":1,"idInTest":1,"iterationInInstance":4,"iterationInScenario":1,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: GoError: browser not found in registry. make sure to set browser type option in scenario definition in order to use the browser module source=console
INFO[0034] {"idInInstance":3,"idInTest":3,"iterationInInstance":3,"iterationInScenario":3,"tags":{"group":"","scenario":"browserLevel"},"metrics":{"tags":{"group":"","scenario":"browserLevel"},"metadata":{}}}: BrowserContext created source=console
data_received........: 0 B 0 B/s
data_sent............: 0 B 0 B/s
iteration_duration...: avg=705.92ms min=400.46ms med=709.23ms max=1.01s p(90)=1s p(95)=1s
iterations...........: 12 0.344638/s
vus..................: 2 min=0 max=2
vus_max..............: 3 min=3 max=3
running (0m34.8s), 0/3 VUs, 12 complete and 0 interrupted iterations
protocolLevel ✓ [======================================] 2 VUs 1s
browserLevel ✓ [======================================] 0/2 VUs 4s
As you can see in output there is vus_max = 3, while both protocol-level and browser-level tests configured to use 2 VUs.
Therefore, at the moment, browser-level tests cannot be combined with protocol-level tests due to this bug in most cases.