I assume all the API plumbing to actually set this is TODO?
I'd rephrase the comments to make it clear that this is the max time before drain, not a hard max cutoff. There's a huge difference between the two and I'm not actually clear on which one is desired (or if both are)

The missing API plumbing pieces might be some sloppiness on my part from managing two branches of this code - let me check that out.

I will rephrase the comment to mention that it is the max time before drain. I want the behavior to enforce a maximum time after which new requests will no longer use this connection. It's ok for existing requests to drain and finish, as long as new ones no longer attempt to use this connection. The idea is to provide a guarantee that after the configured time, all new traffic is flowing through newly created connections, which in practice will be (well, could be) newer upstream backends that all happen to share an IP (think Kubernetes Cluster IPs backed by pods via label selectors). Let me know if that makes sense or if my thinking is misguided.

I think having this cause draining is the most flexible, since it can be coupled with max_stream_duration to ensure that even long lived streams eventually move to the new connection.

Re your example I would imagine that TCP connections wouldn't survive the IP moving from one pod to another since the new upstream would probably respond with a RST if Envoy tried to send it data on a connection established with a previous pod, causing Envoy to establish a new connection? I still think this feature is useful (e.g. when going through a L4 proxy), but not clear to me why this would be required for the IP reuse case.

I should have clarified that the use case is a single IP whose upstream backends change without any of them going unhealthy. Consider a case where there is a Service IP with a label selector that targets healthy pods with version=old. Then, the Service manifest is updated to select on labels version=new instead. Now, any new connection through this IP will reach pods with version=new, but the existing connections to healthy pods with version=old still remain. For the sake of this example, pods with version=old and version=new may coexist for an arbitrary amount of time. The max connection duration feature would help ensure that after a configured amount of time, those old connections would be drained so that all new requests eventually go through new connections.

curious, any reason the upstream can't do its own drain (version=old doing GOAWAY + HTTP/1.1 connection close, nack health)? It'd be a useful feature either way for things like internet based load balancing having limited TTL when DNS can expire, so no objection just curious :-)

The upstream could do its own drain, but I think it's extra useful for the solution to be upstream-agnostic and implemented purely in Envoy.

The exact use case is for canary deployments using two Service IPs, one that represents the canary and one that represents the stable service. We use Envoy's traffic splitting capabilities to slowly move traffic over to the canary IP, and, finally, update the stable IP to point to the canary pods. I think the real solution is to just discover Endpoint objects and push those into Envoy instead of using the abstract Service IP. However, and to your point, it's useful to be able to solve this use case with Service IPs alone.

Please make sure capacity stats are handed correctly. This should decrease the local capacity and possibly trigger prefetching.

Good call, I'll look into this.