Add `--invoke` option to launch a container from the build result #1168

ktock · 2022-06-15T02:58:19Z

This is a PoC of --invoke option to launch a container based on the build result.
Please give me feedbacks on the design of the monitor and CLI design.

This covers PR1 to PR3 of #1104.

PR1
Add possibility to interactively run a process using the NewContainer API after the build has completed. The build will run with the progressbar until completion. Progressbar will finish and container will be launched. Container redirects all stdio and signals. TTY is enabled if user enabled TTY for main process.

PR2
Add "monitor mode" to the interactive process. When running an interactive process, the user can switch between to process io and monitor process io. Similar to QEMU monitor mode. In monitor mode they can issue additional commands. In the very first PR only supported command may be "exit".

PR3
Add "rollback" command to monitor mode. With this command the user can make modifications in the interactive container and when they issue "rollback" command they are brought back to the initial state. Add "reload" command to monitor mode. This will run the build again(now with possibly updated source) and invoke the shell again.

Example:

FROM ghcr.io/stargz-containers/busybox:1.32.0-org
RUN echo hello > /hello

rollback

Ctrl-a-c switches IO between monitor and container with printing Switched IO message.
rollback rollbacks the container by restarting it.

# buildx build --invoke sh /tmp/ctx
WARNING: No output specified with remote driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.5s (5/5) FINISHED                                                  
 => [internal] load build definition from Dockerfile                         0.1s
 => => transferring dockerfile: 111B                                         0.0s
 => [internal] load .dockerignore                                            0.0s
 => => transferring context: 2B                                              0.0s
 => [internal] load metadata for ghcr.io/stargz-containers/busybox:1.32.0-o  0.4s
 => [1/2] FROM ghcr.io/stargz-containers/busybox:1.32.0-org@sha256:bde48e17  0.0s
 => => resolve ghcr.io/stargz-containers/busybox:1.32.0-org@sha256:bde48e17  0.0s
 => CACHED [2/2] RUN echo hello > /hello                                     0.0s
/ # cat /hello
hello
/ # echo hi > /hi
/ # ls
bin    etc    hi     proc   sys    usr
dev    hello  home   root   tmp    var
/ # Switched IO

(buildx) rollback
(buildx) Switched IO

/ # ls
bin    dev    etc    hello  home   proc   root   sys    tmp    usr    var

reload

reload reloads and builds the Dockerfile.

FROM ghcr.io/stargz-containers/busybox:1.32.0-org
RUN echo hello2 > /hello2

(buildx) reload
WARNING: No output specified with remote driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.4s (5/5) FINISHED                                                  
 => [internal] load build definition from Dockerfile                         0.0s
 => => transferring dockerfile: 113B                                         0.0s
 => [internal] load .dockerignore                                            0.1s
 => => transferring context: 2B                                              0.0s
 => [internal] load metadata for ghcr.io/stargz-containers/busybox:1.32.0-o  0.3s
 => [1/2] FROM ghcr.io/stargz-containers/busybox:1.32.0-org@sha256:bde48e17  0.1s
 => => resolve ghcr.io/stargz-containers/busybox:1.32.0-org@sha256:bde48e17  0.0s
 => CACHED [2/2] RUN echo hello2 > /hello2                                   0.0s
(buildx) Switched IO

/ # ls
bin     etc     home    root    tmp     var
dev     hello2  proc    sys     usr
/ # cat /hello2
hello2

exit

exit exits the monitor.

(buildx) exit

long form

Container is configurable through the long form of --invoke.

# buildx build --invoke entrypoint=sh,env=FOO=bar /tmp/ctx
WARNING: No output specified with remote driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.4s (5/5) FINISHED                                                  
 => [internal] load build definition from Dockerfile                         0.1s
 => => transferring dockerfile: 111B                                         0.0s
 => [internal] load .dockerignore                                            0.0s
 => => transferring context: 2B                                              0.0s
 => [internal] load metadata for ghcr.io/stargz-containers/busybox:1.32.0-o  0.3s
 => [1/2] FROM ghcr.io/stargz-containers/busybox:1.32.0-org@sha256:bde48e17  0.0s
 => => resolve ghcr.io/stargz-containers/busybox:1.32.0-org@sha256:bde48e17  0.0s
 => CACHED [2/2] RUN echo hello > /hello                                     0.0s
/ # env | grep FOO
FOO=bar

TODOs

Remaining concerns
- Which sequence should be used to toggle IO? Currently C-a-c is used. Should it be configurable?
- Should container IO be accessible and readable even after it exits?
- Which driver and result should be used when multiple results are provided?
TODOs
- Discuss and determine the design of monitor and CLI
- Add tests

Please tell me if I'm missing anything.

ktock · 2022-06-15T02:58:42Z

cc @tonistiigi

tonistiigi

This looks like a great start 👍

Which sequence should be used to toggle IO? Currently C-a-c is used. Should it be configurable?

I think it is pretty good atm. We should consider also including ctrl-C. Eg. when you are attached to container and press Ctrl-C it could bring you to monitor mode. Pressing it again could give a warning about how to attach back, and another one close the program.

C-a-c atm doesn't show prompt. Need to press enter to get to the prompt.

Maybe for something common like reload we could add a special shortcut in the future.

Should container IO be accessible and readable even after it exits?

You mean stored in some file? I think we can think about that separately together with thinking if build logs need a similar feature.

Which driver and result should be used when multiple results are provided?

Native arch of the first node for selected driver seems like the best option.

For reload, currently it triggers the whole build again. This could be faster by just running a new solve with the same gateway connection (same Build() callback). That would mean that, for example resolving a tag from registry for the base image is not needed between reloads. Or maybe we need two modes of reload.

Let's try to get this in quickly, mark the flags as experimental (maybe BUILDX_EXPERIMENTAL env?), and then start to interate on it instead of trying to get the first PR to be the perfect experience.

tonistiigi · 2022-06-15T05:42:31Z

build/build.go

+// Invoke invokes a build result as a container.
+func Invoke(ctx context.Context, cfg ContainerConfig) error {
+	if cfg.ResultCtx == nil {
+		return fmt.Errorf("result must be provided")


use pkg/errors only in these repos.

Thank you for the comment. fixed this.

tonistiigi · 2022-06-15T05:49:03Z

build/build.go

+// ResultContext is a build result with the client that built it.
+type ResultContext struct {
+	Client   *client.Client
+	SolveOpt client.SolveOpt


Do you even need SolveOpt in here? The whole build is based on Res. If SolveOpt contains output options it may be even wrong to reuse it here.

Thank you for the comment. fixed this.

ktock · 2022-06-17T06:55:39Z

We should consider also including ctrl-C. Eg. when you are attached to container and press Ctrl-C it could bring you to monitor mode. Pressing it again could give a warning about how to attach back, and another one close the program.

Let's work on it in following-up patches. But it looks like it conflicts with the terminal keybind (for sending signals to the container)?

C-a-c atm doesn't show prompt. Need to press enter to get to the prompt.

Fixed to show (buildx) prompt when the IO is swithced to the monitor.
Container doesn't still show the prompt just after C-a-c.
This is because C-a-c just toggle the IO stream but doesn't request the prompt to the attached container.
User need to explicitly send newline or something to get the prompt.

You mean stored in some file? I think we can think about that separately together with thinking if build logs need a similar feature.

Sure.

Native arch of the first node for selected driver seems like the best option.

Current implementation uses the lowest index of the selected drivers. Does this look good?

BUILDX_EXPERIMENTAL

Added this.

tonistiigi

But it looks like it conflicts with the terminal keybind (for sending signals to the container)?

Yes, indeed it seems hard to predict if the signal was for cancelling the process or shell. Maybe if the process dies it goes to the buildx prompt first. We can think about this more later.

User need to explicitly send newline or something to get the prompt.

Could we just print enter for the user. I think we should either print enter or buffer the last line user saw and reprint it again. https://github.com/tonistiigi/vt100 could help if want to figure out what the last tty line is from a stream.

tonistiigi · 2022-06-17T20:31:31Z

commands/build.go

 	err = wrapBuildError(err, false)
 	if err != nil {
 		return err
 	}

+	if in.invoke != "" {
+		if os.Getenv("BUILDX_EXPERIMENTAL") != "1" {


Better to just put the flag definition flags.StringVar(&options.invoke behind env.

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>

ktock · 2022-06-21T01:34:24Z

Could we just print enter for the user.

Fixed to automatically send a newline to the container on C-a-c as it seems the easiest solution as of now.
In the future we can consider richer UI like re-printing the last line.

Better to just put the flag definition flags.StringVar(&options.invoke behind env.

Fixed this.

ktock force-pushed the invoke branch from 42d2536 to 733516c Compare June 15, 2022 03:11

tonistiigi reviewed Jun 15, 2022

View reviewed changes

ktock force-pushed the invoke branch 3 times, most recently from 4233661 to 6998921 Compare June 17, 2022 06:50

ktock marked this pull request as ready for review June 17, 2022 08:47

tonistiigi reviewed Jun 17, 2022

View reviewed changes

Add --invoke option to launch a container from the build result

387e1ec

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>

ktock force-pushed the invoke branch from 6998921 to 387e1ec Compare June 21, 2022 01:32

tonistiigi approved these changes Jun 21, 2022

View reviewed changes

tonistiigi merged commit a8bb25d into docker:master Jun 21, 2022

ktock deleted the invoke branch June 21, 2022 23:54

crazy-max added this to the v0.9.0 milestone Jun 23, 2022

gaocegege mentioned this pull request Jul 13, 2022

feat(CLI): Support debug command tensorchord/envd#124

Open

crazy-max added the area/build label Sep 10, 2022

jedevc added the area/debug label Feb 9, 2023

jedevc mentioned this pull request May 10, 2023

docs: add initial guide about debugging monitor mode #1782

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `--invoke` option to launch a container from the build result #1168

Add `--invoke` option to launch a container from the build result #1168

Uh oh!

ktock commented Jun 15, 2022 •

edited

Loading

Uh oh!

ktock commented Jun 15, 2022

Uh oh!

tonistiigi left a comment

Uh oh!

tonistiigi Jun 15, 2022

Uh oh!

ktock Jun 17, 2022

Uh oh!

tonistiigi Jun 15, 2022

Uh oh!

ktock Jun 17, 2022

Uh oh!

ktock commented Jun 17, 2022

Uh oh!

tonistiigi left a comment

Uh oh!

tonistiigi Jun 17, 2022

Uh oh!

ktock commented Jun 21, 2022

Uh oh!

Uh oh!

Add --invoke option to launch a container from the build result #1168

Add --invoke option to launch a container from the build result #1168

Uh oh!

Conversation

ktock commented Jun 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example:

rollback

reload

exit

long form

TODOs

Uh oh!

ktock commented Jun 15, 2022

Uh oh!

tonistiigi left a comment

Choose a reason for hiding this comment

Uh oh!

tonistiigi Jun 15, 2022

Choose a reason for hiding this comment

Uh oh!

ktock Jun 17, 2022

Choose a reason for hiding this comment

Uh oh!

tonistiigi Jun 15, 2022

Choose a reason for hiding this comment

Uh oh!

ktock Jun 17, 2022

Choose a reason for hiding this comment

Uh oh!

ktock commented Jun 17, 2022

Uh oh!

tonistiigi left a comment

Choose a reason for hiding this comment

Uh oh!

tonistiigi Jun 17, 2022

Choose a reason for hiding this comment

Uh oh!

ktock commented Jun 21, 2022

Uh oh!

Uh oh!

Add `--invoke` option to launch a container from the build result #1168

Add `--invoke` option to launch a container from the build result #1168

ktock commented Jun 15, 2022 •

edited

Loading