Skip to content

Request context canceled during graceful shutdown #3096

@LinEvil

Description

@LinEvil

What happened:

  1. Send a http request, it is slow and takes 5s to return.
  2. Terminate the program by Ctrl+C before the request returns.
  3. Graceful shutdown and the context of the request is canceled.
  4. Throwing context canceled error and exit.

Logs:

INFO msg=[HTTP] server listening on: 127.0.0.1:8000
DEBUG msg=I'm going to sleep for 5 seconds.
INFO msg=[HTTP] server stopping
DEBUG msg=Time's up. Now to check my context.
DEBUG msg=context canceled
INFO msg=watcher's ctx cancel : context canceled

What you expected to happen:

The context of the request should not be canceled, and any request before executing a graceful shutdown should be processed normally unless the request time reaches http.timeout or shutdown time reaches stopTimeout.

How to reproduce it (as minimally and precisely as possible):

# configs/config.yaml
server:
  http:
    addr: 127.0.0.1:8000
    timeout: 8s
// internal/service/greeter.go
func (s *GreeterService) SayHello(ctx context.Context, in *v1.HelloRequest) (*v1.HelloReply, error) {
	s.log.WithContext(ctx).Infof("SayHello Received: %v", in.GetName())
	s.log.WithContext(ctx).Debug("I'm going to sleep for 5 seconds.")
	time.Sleep(5 * time.Second)	// press Ctrl+C
	s.log.WithContext(ctx).Debug("Time's up. Now to check my context.")
	if ctx.Err() == context.Canceled {
		s.log.WithContext(ctx).Debug("context canceled")
		return nil, ctx.Err()
	}
	return &v1.HelloReply{Message: "Hello " + in.GetName()}, nil
}

Anything else we need to know?:

The issue is similar to #1856, which was fixed in v2.2.1 (#1873 #1895).
My case works fine between v2.2.1 and v2.5.2

Since v2.5.3, the feature #2403 changed srv.Start(context), which may cause the current issue.
I tried rolling back that line and the problem was fixed, but I'm not sure if this is right.

Environment:

  • Kratos version (use kratos -v): v2.7.1
  • Go version (use go version): go1.21.4
  • OS (e.g: cat /etc/os-release): Windows
  • Others:

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions