Skip to content

x/net/http2: client sends too many pings for gRPC servers since v0.31.0 #70575

@RemiBou

Description

@RemiBou

Go version

1.23.3

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT='boringcrypto'
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/.gvm/pkgsets/go1.23.3/global/pkg/mod'
GONOPROXY='none'
GONOSUMDB='...'
GOOS='linux'
GOPATH='/root/.gvm/pkgsets/go1.23.3/global'
GOPRIVATE='...'
GOPROXY='...'
GOROOT='/root/.gvm/gos/go1.23.3'
GOSUMDB='off'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/root/.gvm/gos/go1.23.3/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.3'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='.../aarch64-linux-musl-cross/bin/aarch64-linux-musl-gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='.../go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1336983329=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We use traefik as a library for our service mesh.

What did you see happen?

Since upgrading x/net to 0.31.0 we have this error appearing on our instance that is deployed as a sidecar of a grpc java server

2024/11/26 10:32:41 httputil: ReverseProxy read error during body copy: http2: server sent GOAWAY and closed the connection; LastStreamID=561, ErrCode=ENHANCE_YOUR_CALM, debug="too_many_pings"

Downgrading net to 0.30 fixes the issues

I think it might be due to the new ping added here

golang/net@f35fec9#diff-e9bd9b4a514c2960ad85405e4a827d83c2decaaac70e9e0158a59db2d05807a7R3149 by @neild

You can find the framer logs here (http2debug=2)
framerlog.log

This message is returned by grpc java here

https://github.com/grpc/grpc-java/blob/a79982c7fded85ac6f53896f045e959b6cbd7c43/netty/src/main/java/io/grpc/netty/NettyServerHandler.java#L983

Which runs this function https://github.com/grpc/grpc-java/blob/master/core/src/main/java/io/grpc/internal/KeepAliveEnforcer.java#L57

This basically refuses a ping if one was sent less than 2 hour before

What did you expect to see?

Stream to be kept and not reset

Metadata

Metadata

Assignees

No one assigned

    Labels

    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions