Integrate Binary provisioning #4671

pablochacin · 2025-04-03T18:36:32Z

What?

Integrate into k6 the logic for dynamically provision a k6 custom binary that includes the dependencies (extensions) required by a test.

This change set introduces the launcher as the new starting point for executing k6. The launcher checks if a binary provisioning is enabled and if it is required to execute the command. If not, it delegates the executing to the regular execution entry point.

Binary provisioning is required if the command processes a script or archive (run, archive, ...) and the input has a dependency that cannot be resolved with the current k6 binary. This is resolved by the k6deps library that scans the input for imports of k6 extensions.

If a new binary is required, it is downloaded from the build service using the k6provider library. This requires a cloud token for authentication. Binaries are stored in a local cache, so subsequent requests will be served from the local file system instead of downloading it.

Note: This is a work in progress. It is fully functional, but a few tasks are still pending:

TODO:

Minimize the dependency diff, precisely cobra and Prometheus dependencies, which should be defined by k6

Also, some features are intended for follow-up PRs:

FOLLOW-UP:

Reuse existing logic for reading k6 config
Handle stdin as input (this requires creating a tmp file, and it's not clear how to do this early in the execution path)
handle URLs as input (same as above and also has some added complexity of identifying the URL of the input as other arguments may also be URLs)
When checking if binary provisioning is required, consider case when binary has built in extensions
Better visibility for cache usage & management

Why?

We'd like k6 to be able to run scripts with extensions without recompiling the binary

Checklist

I have performed a self-review of my code.
I have commented on my code, particularly in hard-to-understand areas.
I have added tests for my changes.
I have run linter and tests locally (make check) and all pass.

Checklist: Documentation (only for k6 maintainers and if relevant)

Please do not merge this PR until the following items are filled out.

I have added the correct milestone and labels to the PR.
I have updated the release notes: link
I have updated or added an issue to the k6-documentation: grafana/k6-docs#NUMBER if applicable
I have updated or added an issue to the TypeScript definitions: grafana/k6-DefinitelyTyped#NUMBER if applicable

Related PR(s)/Issue(s)

Decision mechanics if custom k6 build required

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

joanlopez · 2025-04-24T14:42:36Z

internal/cmd/launcher.go

+		return // this is required for testing
+	}
+
+	// if the command does not have dependencies or a custom build


Suggested change

// if the command does not have dependencies or a custom build

// if the command does not have dependencies nor a custom build is required

joanlopez · 2025-04-24T14:44:41Z

internal/cmd/launcher.go

+	cmd.Stdin = os.Stdin
+
+	// Copy environment variables to the k6 process and skip binary provisioning feature flag to disable it.
+	// If not disabled, then the executed k6 binary would enter an infinite loop, where it continuously


Can we add a // TODO here, please? I guess this shouldn't be a problem anymore in the future. Right?

I think it will always be required/desirable to signal the subprocess that the binary provisioning is not required to avoid the overhead of reanalyzing the dependencies.

joanlopez · 2025-04-24T14:46:16Z

internal/cmd/launcher.go

+	// process the input script, detect dependencies, and retrigger provisioning.
+	env := []string{}
+	for k, v := range gs.Env {
+		if k == "K6_BINARY_PROVISIONING" {


I'd say I have seen this literal string at least a couple of times along this PR, so I'd prefer if we can move this into a const, so prevent mistakes in the future, in case it changes, possible typos, etc.

joanlopez · 2025-04-24T14:52:37Z

internal/cmd/launcher.go

+}
+
+// isScriptRequired searches for the command and returns a boolean indicating if it is required to pass a script or not
+func isScriptRequired(args []string) bool {


In case we run k6 cloud run with --local-execution, I think this should return false, as I guess it's equivalent to k6 run, isn't it? 🤔
Or, do we intentionally want to run a custom binary locally in such case?

In any case, I'd suggest to add that case to TestIsScriptRequired, so we leave it explicit. Perhaps even accompanied with a code comment, as in my opinion isn't obvious.

could run --local-execution is expected to run with binary provisioning. We are not making a difference with cloud run.

joanlopez · 2025-04-24T14:58:43Z

internal/cmd/launcher.go

+		if strings.HasSuffix(arg, ".js") ||
+			strings.HasSuffix(arg, ".tar") ||
+			strings.HasSuffix(arg, ".ts") {
+			return arg
+		}


I still think this isn't a good idea because:

k6 normally works despite of the file extension.

If I have binary provisioning enabled, it breaks the execution flow even when there's no dependency to analyze.

The error is confusing, because it says: error="script not found", while arguably the problem is that the file extension isn't supported/recognized as a file script, or something along those lines.

In my opinion, we should try to fix at least 2 and 3 before merging PR. 1 is something we can work later, I guess - I'd appreciate if we can create and issue for that, so we don't forget about remaining work, please.

My understanding is that we accepted this as a limitation for the preview and document it as a known issue..

Addressing this issue in the current logic is complex and probably will still fail in some cases.

It is documented here: #4727

pablochacin · 2025-04-24T15:04:31Z

@joanlopez Regarding the issue you mention with cancelling a cloud execution in the subprocess, the fact you see this message

INFO[0014] Stopping cloud test run in response to signal...  sig=interrupt

Means that the subprocess is actually being properly stopped by the parent. Otherwise, you woulnd' t see that message.

My guess is that you still see the process because as this message says, it is waiting for the cloud.:

INFO[0014] Successfully sent signal to stop the cloud test, now waiting for it to actually stop...

I'm not sure what the expectation is in this case.

Should the parent wait for the children? to finish. I'm not 100% sure, but I would expect this to be the case. So I'm not sure why the parent finishes first

cc @codebien

joanlopez · 2025-04-24T15:15:58Z

I'm not 100% sure, but I would expect this to be the case.

Same here, that's why I asked.

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

codebien · 2025-04-24T15:38:08Z

Should the parent wait for the children? to finish. I'm not 100% sure, but I would expect this to be the case. So I'm not sure why the parent finishes first

It should be related to the signal trap enabled by the subprocess where the launcher doesn't have any because it doesn't reach that part of the code.

pablochacin · 2025-04-24T15:38:45Z

I'm not 100% sure, but I would expect this to be the case.

Same here, that's why I asked.

@codebien commented that the cause is we are not handling the signal in the parent, so it is ending prematurely.
I will try to add the signal handling to prevent this.

oleiade

LGMT (with my current level of context on the topic) 👍🏻 🚀

Once the signals handling PR is merged, I think this is good to go. I share some of the observations already made, but considering we already have most of the points raised on the radar documented as issues to be tackled in further iterations 👍🏻 on my end.

Really happy to see this (tentatively) passing the finish line 🏁 Grateful for the work you put in @pablochacin 👏🏻

oleiade · 2025-04-29T12:21:18Z

internal/cmd/launcher.go

+
+	// provision generates a custom binary from the received list of dependencies
+	// with their constrains, and it returns an executor that satisfies them.
+	provision func(*state.GlobalState, k6deps.Dependencies) (commandExecutor, error)


Nit (feel free to ignore/dismiss): although it is not strictly necessary here, I tend to prefer, when possible, defining and implementing an interface (Provisioner?), instead of a callback. The reason is ease of debugging and codebase navigation in IDEs. It's much easier to go to definition of an interface and ask for implementations, than to track down where the concrete callback instance is defined and passed down. Sometimes it also makes testing more straightforward.

No arguing on my end, really just a suggestion and expressing a preference, I'm okay with it as-is 👍

@pablochacin I agree with @oleiade, at least it should be worth to have a reflection/attempt on refactoring it. Can you open and add an issue to the stability epic, please?

codebien · 2025-04-29T13:57:37Z

@pablochacin additional reason for #4737

Help for subcommands seems an additional issue with the current solution:

K6_BINARY_PROVISIONING=1 k6 cloud run --help
ERRO[0000] Binary provisioning is enabled but it failed to analyze the dependencies. Please, make sure to report this issue by opening a bug report.  error="script not found"

* handle signals Signed-off-by: Pablo Chacin <pablochacin@gmail.com> * attend review comments Signed-off-by: Pablo Chacin <pablochacin@gmail.com> --------- Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

inancgumus

LGTM 👍 Nice work.

pablochacin · 2025-04-30T08:57:12Z

@codebien I implemented a workaround fo the issue with the help commands here #4753
But agree this approach is not sustainable as we are pre-parsing the CLI instead of relying on k6 command parsing logic.

Migrate from k6provider to k6 dir for builds' cache --------- Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

This was referenced Apr 3, 2025

WIP: Binary provisioning poc #4632

Closed

[WIP] k6exec into k6 #4626

Closed

pablochacin self-assigned this Apr 3, 2025

pablochacin changed the title ~~Feat/binary provisioning~~ Integrage Binary provisioning Apr 3, 2025

pablochacin force-pushed the feat/binary-provisioning branch 5 times, most recently from c1d3f2d to 42392ac Compare April 7, 2025 15:34

olegbespalov and others added 17 commits April 8, 2025 16:31

k6exec naive usage

3b52a41

Decision mechanics if custom k6 build required

Use k6's config file

cebb21e

Better handling of script args

6b9282c

Libraries updates and more simplifications of the logic

09021bf

simplify code structure

890fb4b

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

remove redundant ExecuteWithGlobalState method

3cfd619

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

simplify launcher's logic

96ac5d7

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

improve visibility of binary provisioning process

4961632

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

simplify analize logic

458cfe7

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

refactor to facilitate testing

4d40785

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

remove unnecessary error formatting logic

3fc4ebb

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

refactor os exit to improve testability

d0e8faf

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

add tests

92d6012

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

fix validation of provisioning required

cb9b3f5

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

fix linter issues

fe0c0d9

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

fix logic to check if provisioning is required

d5eae2b

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

improve comments and fix log messages

4c111b7

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

pablochacin force-pushed the feat/binary-provisioning branch from b521a97 to 4c111b7 Compare April 8, 2025 14:31

update k6deps

524a232

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

pablochacin marked this pull request as ready for review April 9, 2025 08:57

pablochacin requested a review from a team as a code owner April 9, 2025 08:57

joanlopez reviewed Apr 24, 2025

View reviewed changes

Relax the feature flag conditions

b5e77b3

joanlopez reviewed Apr 24, 2025

View reviewed changes

address review comments

0d1951e

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

oleiade previously approved these changes Apr 29, 2025

View reviewed changes

handle signals (#4740)

bdfea21

* handle signals Signed-off-by: Pablo Chacin <pablochacin@gmail.com> * attend review comments Signed-off-by: Pablo Chacin <pablochacin@gmail.com> --------- Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

codebien dismissed oleiade’s stale review via bdfea21 April 29, 2025 14:19

inancgumus previously approved these changes Apr 29, 2025

View reviewed changes

Binary provisioning/add cache dir flag (#4743)

7034e39

Migrate from k6provider to k6 dir for builds' cache --------- Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

codebien dismissed inancgumus’s stale review via 7034e39 April 30, 2025 09:59

pablochacin and others added 4 commits April 30, 2025 12:00

Skip binary provisioning for help commands (#4753)

eed742e

Signed-off-by: Pablo Chacin <pablochacin@gmail.com>

Merge branch 'master' into feat/binary-provisioning

9968498

Merge branch 'master' into feat/binary-provisioning

143403b

Merge branch 'master' into feat/binary-provisioning

ddb7836

codebien approved these changes May 2, 2025

View reviewed changes

andrewslotin approved these changes May 2, 2025

View reviewed changes

Merge branch 'master' into feat/binary-provisioning

7f3b93d

oleiade approved these changes May 2, 2025

View reviewed changes

codebien merged commit a485f1b into master May 2, 2025
30 of 31 checks passed

codebien deleted the feat/binary-provisioning branch May 2, 2025 11:14

pablochacin mentioned this pull request Jun 19, 2025

Integrate binary provisioning into k6 root command #4801

Merged

8 tasks

	// if the command does not have dependencies or a custom build
	// if the command does not have dependencies nor a custom build is required

Integrate Binary provisioning #4671

Integrate Binary provisioning #4671

Uh oh!

Conversation

pablochacin commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

Checklist

Checklist: Documentation (only for k6 maintainers and if relevant)

Related PR(s)/Issue(s)

Uh oh!

joanlopez Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joanlopez Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

pablochacin Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

joanlopez Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

joanlopez Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

pablochacin Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

joanlopez Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablochacin Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pablochacin commented Apr 24, 2025

Uh oh!

joanlopez commented Apr 24, 2025

Uh oh!

codebien commented Apr 24, 2025

Uh oh!

pablochacin commented Apr 24, 2025

Uh oh!

oleiade left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oleiade Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

codebien Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codebien commented Apr 29, 2025

Uh oh!

inancgumus left a comment

Choose a reason for hiding this comment

Uh oh!

pablochacin commented Apr 30, 2025

Uh oh!

Uh oh!

Uh oh!

pablochacin commented Apr 3, 2025 •

edited

Loading

joanlopez Apr 24, 2025 •

edited

Loading

joanlopez Apr 24, 2025 •

edited

Loading

pablochacin Apr 24, 2025 •

edited

Loading

oleiade left a comment •

edited

Loading

codebien Apr 30, 2025 •

edited

Loading