Skip to content

Retry policy without perTryTimeout in VirtualService makes grpc request timeout unexpectlly #40299

@suninuni

Description

@suninuni

Bug Description

Retry policy without perTryTimeout in VirtualService will cause some UT request for grpc.

image

The default retry policy in RDS is

"route": {
    "cluster": "outbound|1600||grpc-service.default.svc.cluster.local",
    "timeout": "0s",
    "retryPolicy": {
        "retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
        "numRetries": 2,
        "retryHostPredicate": [
            {
                "name": "envoy.retry_host_predicates.previous_hosts"
            }
        ],
        "hostSelectionRetryMaxAttempts": "5",
        "retriableStatusCodes": [
            503
        ]
    },
    "maxStreamDuration": {
        "maxStreamDuration": "0s",
        "grpcTimeoutHeaderMax": "0s"
    }
}

After I apply the VirtualService

spec:
  hosts:
  - grpc-service
  http:
  - retries:
      attempts: 1
      retryOn: unavailable,cancelled
    route:
    - destination:
        host: grpc-service
        port:
          number: 1600

The RDS config changes to

"route": {
    "cluster": "outbound|1600||grpc-service.default.svc.cluster.local",
    "timeout": "0s",
    "retryPolicy": {
        "retryOn": "unavailable,cancelled",
        "numRetries": 1,
        "retryHostPredicate": [
            {
                "name": "envoy.retry_host_predicates.previous_hosts"
            }
        ],
        "hostSelectionRetryMaxAttempts": "5"
    },
    "maxGrpcTimeout": "0s"
}

It deleted maxStreamDuration and added maxGrpcTimeout, I can't confirm which configuration is causing those UT requests.

But I try to modify the defaultHttpRetryPolicy in mesh config, it will not cause these promblem.

    defaultHttpRetryPolicy:
      retryOn: unavailable
      attempts: 1
"route": {
    "cluster": "outbound|1600||grpc-service.default.svc.cluster.local",
    "timeout": "0s",
    "retryPolicy": {
        "retryOn": "unavailable",
        "numRetries": 1,
        "retryHostPredicate": [
            {
                "name": "envoy.retry_host_predicates.previous_hosts"
            }
        ],
        "hostSelectionRetryMaxAttempts": "5"
    },
    "maxStreamDuration": {
        "maxStreamDuration": "0s",
        "grpcTimeoutHeaderMax": "0s"
    }
}

Version

$ istioctl version
client version: 1.13.1
control plane version: 1.13.1
data plane version: 1.13.1 (135 proxies)

Additional Information

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions