@Kump3r @IvanChalukov I'll wait for you guys to review and test before merging.

Got thinking a bit more last night, wondering if there's a way to do this without having to track the IO's ourselves. Tried doing this:

diff --git a/worker/runtime/container.go b/worker/runtime/container.go
index 0c533811a..6e090f183 100644
--- a/worker/runtime/container.go
+++ b/worker/runtime/container.go
@@ -186,9 +186,18 @@ func (c *Container) Attach(pid string, processIO garden.ProcessIO) (process gard
                return nil, fmt.Errorf("task attach: %w", err)
        }

+       prevIO := task.IO()
        cioOpts := containerdCIO(processIO, false)
+       var attach cio.Attach = func(f *cio.FIFOSet) (cio.IO, error) {
+               newIO, err := cio.NewAttach(cioOpts...)(f)
+               if prevIO != nil {
+                       prevIO.Cancel()
+                       prevIO.Close()
+               }
+               return newIO, err
+       }

-       proc, err := task.LoadProcess(ctx, pid, cio.NewAttach(cioOpts...))
+       proc, err := task.LoadProcess(ctx, pid, attach)
        if err != nil {
                return nil, fmt.Errorf("load proc: %w", err)
        }

Sadly, the cio.IO that the task returns at this point in the code is a new cio.IO that just points to the FIFO files. It doesn't contain the previous cio.IO that has the files open. Really seems like containerd doesn't give us a way to lookup the previous/existing cio.IO at all.

Unless someone figures out a way to lookup existing cio.IO's for a given containerd Task, what we're doing is probably the easiest thing for us to do.

Got thinking a bit more last night, wondering if there's a way to do this without having to track the IO's ourselves.

We though and tested something similar, although to no effect. Sadly seems like containerd does not keep any metadata information about previous logs anywhere. Possibly in the future, should they implement such a feature, we could revisit. For now I think we all reach the same conclusion/direction as to what you did. Although yours is 10x better, since I didn't even see the backend for garden server 🤣
We will invest some time in reviewing, although I think we might lack some knowledge to provide adequate suggestions on improvements. In any case I will test your solution in our environments to ensure it solves the underlying issue and does not bring weird regressions/bugs.

Hi @taylorsilva, I've run the following tests:

Local tests

Setup with 1 worker and 1 web instance:

1. Restarting the worker

• No logs were lost after the worker restarted.
• fly watch did not lose logs after the worker restarted.
• fly intercept maintained its connection after the worker restarted.

2. Restarting the web

• Logs were missing only during the web restart(as expected), but resumed streaming as expected once the web came back up.
• fly watch behaved the same as the UI: logs were missing only during the web restart and resumed afterward.
• fly intercept dropped its connection during the web restart, as expected.

Bosh environment

It looks like there's a regression when running with 2 web nodes and 2 workers (the number of workers doesn't seem to matter). Both errors appears after sequential web restarts(trying to simulate rolling update of web workers)

1. Fails in task.Exec(..) in container Run() method and kills container.

• After the first web restart, the worker switched over to the second web and streaming continue as expected.
• But after restarting the second web, the worker switched back to the first and then failed with:

{"timestamp":"2025-07-15T12:34:36.575461374Z","level":"error","source":"worker","message":"worker.garden.garden-server.run.failed","data":{"error":"task exec: id task: already exists","handle":"163013a7-1a21-4920-9449-5f2f9e956fea","session":"1.2.127"}}

2. Fails in task.LoadProcess(..) in container Attach() method and kills container.
   Couldn’t find a clear pattern, but during the restart of the first web, sometimes this was thrown:

wait for process completion: backend error: Exit status: 500, message: {"Type":"","Message":"load proc: failed to open stdin fifo: error creating fifo /run/containerd/fifo/4277305638/task-stdin: no such file or directory","Handle":"","ProcessID":"","Binary":""}

Summary

• Single web + single worker behaves correctly under both web and worker restarts.
• With multiple web nodes, there are regressions related to task ID reuse and missing FIFOs after sequential web restarts.

Did you try running with 2 web instances? I’ll dig into this further to see if I can pinpoint the issue.

I do not have time to test, but shooting a suspicion, feel free to disregard if it BS.
Is it possible that during bosh web process restart, the in memory data for the original task is not flushed. Then the second web attaches and closes the streams. But as soon as the second web shutdowns. The original web 1 tries to attach with the old fifo/streams?

I haven't tried with two web instances. I'll see how I can reproduce that locally.

Not sure why that first error is happening. The second error, no such file or directory, I saw happen when we detach from the FIFO's before attaching the new one's. Hopefully some more debugging will explain why this is happening.

I'll investigate and share what I find here.

I was able to reproduce the issue locally and fix it. I'm going to try adding an integration test for this as well since a lot of users have two web nodes in their Concourse clusters.

The underlying error for both scenario's Ivan reported was:

load proc: failed to open stdin fifo: error creating fifo /run/containerd/fifo/1731945221/task-stdin: no such file or directory

You only saw this in the worker's logs and not in the web ui. This happens because the web node always tries attaching to a container first and then falls back to initializing a new task if it gets an error trying to attach. The reason for this has to do with the Garden API and keeping compatibility with the Guardian backend.

Anyways, figured out we just need to not fully close out the previous IO's. It seems doing that would signal to containerd that it can delete the tasks's FIFO files. At this point it seems things could get race-y. I'm guessing this is why Ivan would sometimes see the failed to open stdin fifo.

The only concern I have now is that I'm not sure if we're leaking anything by not calling Close() on the previous CIO's.
I've confirmed that the FIFO files are deleted once the task is done running, so it doesn't seem like we're leaking anything. I used lsof and did multiple rolling restarts of the web nodes to see if I saw an increase in lsof's output, but didn't, it stayed stable. So I don't think we're leaking anything by not calling Close() on the previous CIO.

Ivan, if you could do your tests one more time that would be great. You caught a good bug here!

That all makes sense, thanks for digging into it!

You only saw this in the worker logs and not in the web UI.

Actually, I observed those errors in both the worker logs and the UI logs:

That said, I ran the same tests again, both locally and in a BOSH environment, and I can confirm that after the web restarts, there are no errors in either the worker or UI logs.

I also kept a close eye on the FIFO files, and they’re being cleaned up exactly as they should after reattaching to the task. So Containerd isn’t leaking anything from my perspective - it’s correctly cleaning up the FIFOs even without explicitly calling Close() on the previous CIOs.

Really nice catch on the need to avoid fully closing the previous CIOs. Looks like that solved the issue cleanly.

I've added an integration test that brings up two web nodes and tests that build logs are not lost after multiple rolling restarts of the web nodes. It checks for the first few log lines "Hello {1...3}" and some of the last few "Hello {110...112}" as verification that logs continue to stream with a fly watch command.