Skip to content

Panic in the Garden #1501

@tiredpixel

Description

@tiredpixel

Using the same environment as detailed in #1424 (I don't know if it's related), running Concourse 3.4.0 from Docker images, workers stall after an hour or two on busy pipelines with the following:

2017/08/18 12:34:43 http: panic serving 127.0.0.1:56260: runtime error: invalid memory address or nil pointer dereference
goroutine 271183 [running]:
net/http.(*conn).serve.func1(0xc4201585a0)
        /usr/local/go/src/net/http/server.go:1721 +0xd0
panic(0xf913c0, 0xfc9e090)
        /usr/local/go/src/runtime/panic.go:489 +0x2cf
code.cloudfoundry.org/garden/server/streamer.(*Streamer).serve(0xc420432db0, 0xc4202fd727, 0x4, 0xfcc16e0, 0xc42000f128, 0x1)
        /tmp/build/9674af12/concourse/src/code.cloudfoundry.org/garden/server/streamer/streamer.go:70 +0x64
code.cloudfoundry.org/garden/server/streamer.(*Streamer).ServeStderr(0xc420432db0, 0xc4202fd727, 0x4, 0xfcc16e0, 0xc42000f128)
        /tmp/build/9674af12/concourse/src/code.cloudfoundry.org/garden/server/streamer/streamer.go:64 +0x5c
code.cloudfoundry.org/garden/server/streamer.(*Streamer).ServeStderr-fm(0xc4202fd727, 0x4, 0xfcc16e0, 0xc42000f128)
        /tmp/build/9674af12/concourse/src/code.cloudfoundry.org/garden/server/server.go:93 +0x52
code.cloudfoundry.org/garden/server/streamer.HandlerFunc.ServeHTTP(0xc420422a20, 0xfccee60, 0xc4228b1340, 0xc420d65100)
        /tmp/build/9674af12/concourse/src/code.cloudfoundry.org/garden/server/streamer/http_handling.go:21 +0x144
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc420418fa0, 0xfccee60, 0xc4228b1340, 0xc420d65100)
        /tmp/build/9674af12/concourse/src/github.com/bmizerany/pat/mux.go:117 +0x63f
code.cloudfoundry.org/garden/server.New.func1(0xfccee60, 0xc4228b1340, 0xc420d65100)
        /tmp/build/9674af12/concourse/src/code.cloudfoundry.org/garden/server/server.go:112 +0x4d
net/http.HandlerFunc.ServeHTTP(0xc420419060, 0xfccee60, 0xc4228b1340, 0xc420d65100)
        /usr/local/go/src/net/http/server.go:1942 +0x44
net/http.serverHandler.ServeHTTP(0xc4200976b0, 0xfccee60, 0xc4228b1340, 0xc420d65100)
        /usr/local/go/src/net/http/server.go:2568 +0x92
net/http.(*conn).serve(0xc4201585a0, 0xfcd0060, 0xc422065c00)
        /usr/local/go/src/net/http/server.go:1825 +0x612
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2668 +0x2ce
NAME                                        STATUS    AGE       VERSION   EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION
ip-IP.eu-west-1.compute.internal   Ready     4h        v1.7.0    IP   Ubuntu 16.04.3 LTS   4.10.0-28-generic
ip-IP.eu-west-1.compute.internal   Ready     17d       v1.7.0    IP   Ubuntu 16.04.3 LTS   4.10.0-28-generic
ip-IP.eu-west-1.compute.internal   Ready     4h        v1.7.0    IP   Ubuntu 16.04.3 LTS   4.10.0-28-generic

Despite the various issues with 3.x, I've never seen this particular problem prior to Concourse 3.4.0 upgrade.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions