Locality based routing not working for istio-proxy to istio-egressgateway traffic... #39792
Description
Bug Description
When our traffic routes out from an istio-proxy sidecar through an istio-egressgateway pod, we want the traffic to stay within-zone whenever possible so that we can avoid paying cross-AZ network costs. For setting up the EgressGateways, we followed the docs at https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway/ (with a minor change, I'll describe it in the "Additional Information" section below).
We have other high volume services within the mesh that use locality-aware routing to great success, so we do understand that it works fundamentally within the mesh. I can confirm that it works when we route traffic from an istio-ingressgateway to a pod with an istio-proxy sidecar. It just doesn't seem to work in the reverse flow out of an egressgateway.
I've posted on Slack (https://istio.slack.com/archives/C37A4KAAD/p1657034565248319, https://istio.slack.com/archives/C37A4KAAD/p1656698370327109) and also dug through the discuss.istio.io page for help on this but I haven't found the answer yet.
My concern here is that this may simply not be a supported thing to do - or if it is, there must be something in the documentation we're missing or not understanding..
Version
$ istioctl version
client version: 1.13.3
control plane version: 1.14.0
data plane version: 1.14.0 (19 proxies)
$ kubectl version --short
Client Version: v1.23.5
Server Version: v1.22.9-eks-a64ea69
### Additional Information
Other than a fairly standard Istio installation, here are the resources we've configured to try to make the egress routing work. We followed the docs at https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway/ pretty closely, however we made one change to reconfigure plaintext HTTP traffic (`http://httpbin.org`) to go out to the target destination via HTTPS:
## `Gateway: istio-egressgateway`
```yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
generation: 3
name: istio-egressgateway
namespace: istio-system
spec:
selector:
istio: egressgateway
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP
- hosts:
- '*'
port:
name: tls
number: 443
protocol: TLS
tls:
mode: PASSTHROUGH
DestinationRule: istio-egressgateway
We created a DestinationRule for the egressgateway itself (traffic from the mesh -> gateway) to try to control the outlier detection:
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: istio-egressgateway
namespace: istio-system
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
trafficPolicy:
connectionPool:
http:
maxRequestsPerConnection: 100
outlierDetection:
baseEjectionTime: 30s
consecutiveGatewayErrors: 7
interval: 10sServiceEntry: httpbin-org
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
generation: 1
name: httpbin-org
namespace: istio-system
spec:
hosts:
- httpbin.org
ports:
- name: http
number: 80
protocol: HTTP
targetPort: 443
- name: tls
number: 443
protocol: TLS
resolution: DNSDestinationRule: httpbin-org
This DestinationRule is used to reconfigure traffic targeting port 80 to re-route to port 443... it should only apply on the egressgateway where the final traffic out to the real destination is happening:
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: httpbin-org
namespace: istio-system
spec:
host: httpbin.org
trafficPolicy:
portLevelSettings:
- port:
number: 80
tls:
mode: SIMPLEVirtualService: httpbin-org
Finally here's the VirtualService:
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: httpbin-org
namespace: istio-system
spec:
gateways:
- istio-egressgateway
- mesh
hosts:
- httpbin.org
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
port:
number: 80
weight: 100
- match:
- gateways:
- istio-egressgateway
port: 80
route:
- destination:
host: httpbin.org
port:
number: 80
weight: 100
tls:
- match:
- gateways:
- mesh
port: 443
sniHosts:
- httpbin.org
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
port:
number: 443
- match:
- gateways:
- istio-egressgateway
port: 443
sniHosts:
- httpbin.org
route:
- destination:
host: httpbin.org
port:
number: 443
weight: 100Logs from Requests
Ok, with this in mind, I launched a debug pod with an istio-proxy sidecar and I start curl'ling httpbin.org/ip and we can see that the upstream_host is fairly randomly selected:
$ k -n matt-test logs debug-diranged-8584a8da -c istio-proxy | grep httpbin | grep upstream | jq .upstream_host
"10.32.29.129:8080"
"10.32.28.194:8080"
"10.32.28.194:8080"
"10.32.28.83:8080"
"10.32.28.194:8080"
"10.32.29.129:8080"
"10.32.28.194:8080"
"10.32.29.129:8080"
"10.32.28.83:8080"
"10.32.29.129:8080"
"10.32.29.129:8080"
"10.32.28.83:8080"
"10.32.28.83:8080"
"10.32.28.83:8080"
"10.32.28.194:8080"
The logs on the istio-egressgateway containers are spread out between the containers too:
[pod/istio-egressgateway-57d76dddf9-948t2/istio-proxy] {"response_code_details":"via_upstream","request_id":"b4532f17-d23c-985e-a447-406204fe96a8","connection_termination_details":null,"upstream_transport_failure_reason":null,"requested_server_name":null,"bytes_sent":57,"upstream_local_address":"10.32.25.195:44082","upstream_host":"35.173.123.14:443","upstream_service_time":"66","user_agent":"curl/7.79.1","start_time":"2022-07-05T16:27:42.040Z","downstream_local_address":"10.32.25.195:8080","grpc_status":"Unknown","authorization":null,"response_code":200,"response_flags":"-","path":"/ip","method":"GET","protocol":"HTTP/2","downstream_remote_address":"100.64.110.3:0","duration":67,"authority":"httpbin.org","upstream_cluster":"outbound|80||httpbin.org","route_name":null,"x_forwarded_for":"100.64.110.3,100.64.110.3","bytes_received":0}
[pod/istio-egressgateway-57d76dddf9-948t2/istio-proxy] {"authority":"httpbin.org","connection_termination_details":null,"response_code":200,"request_id":"de431eca-4bde-9cba-bd0e-1c1540d49d0b","upstream_transport_failure_reason":null,"downstream_remote_address":"100.64.110.3:0","path":"/ip","bytes_received":0,"upstream_service_time":"69","downstream_local_address":"10.32.25.195:8080","route_name":null,"method":"GET","protocol":"HTTP/2","user_agent":"curl/7.79.1","x_forwarded_for":"100.64.110.3,100.64.110.3","duration":69,"upstream_local_address":"10.32.25.195:56574","upstream_host":"18.215.122.215:443","grpc_status":"Unknown","response_flags":"-","authorization":null,"response_code_details":"via_upstream","requested_server_name":null,"start_time":"2022-07-05T16:27:44.116Z","bytes_sent":57,"upstream_cluster":"outbound|80||httpbin.org"}
[pod/istio-egressgateway-57d76dddf9-h5lnl/istio-proxy] {"authority":"httpbin.org","connection_termination_details":null,"response_code_details":"via_upstream","route_name":null,"path":"/ip","upstream_local_address":"10.32.29.2:39180","request_id":"93d805d9-f3ea-9cd6-aecb-c01661751081","grpc_status":"Unknown","bytes_received":0,"upstream_host":"3.94.154.124:443","user_agent":"curl/7.79.1","x_forwarded_for":"100.64.110.3,100.64.110.3","authorization":null,"method":"GET","downstream_remote_address":"100.64.110.3:0","protocol":"HTTP/2","start_time":"2022-07-05T16:27:46.195Z","response_code":200,"duration":71,"bytes_sent":59,"downstream_local_address":"10.32.29.2:8080","response_flags":"-","upstream_service_time":"70","requested_server_name":null,"upstream_transport_failure_reason":null,"upstream_cluster":"outbound|80||httpbin.org"}
[pod/istio-egressgateway-57d76dddf9-96gk9/istio-proxy] {"authorization":null,"path":"/ip","response_code_details":"via_upstream","method":"GET","x_forwarded_for":"100.64.110.3,100.64.110.3","authority":"httpbin.org","bytes_received":0,"protocol":"HTTP/2","user_agent":"curl/7.79.1","response_code":200,"duration":69,"start_time":"2022-07-05T16:27:48.274Z","route_name":null,"upstream_transport_failure_reason":null,"upstream_local_address":"10.32.28.194:38612","requested_server_name":null,"downstream_remote_address":"100.64.110.3:0","request_id":"2b43161b-fe06-99cf-b228-cfa368e965f9","upstream_service_time":"69","upstream_cluster":"outbound|80||httpbin.org","bytes_sent":59,"upstream_host":"3.94.154.124:443","connection_termination_details":null,"grpc_status":"Unknown","response_flags":"-","downstream_local_address":"10.32.28.194:8080"}
[pod/istio-egressgateway-57d76dddf9-948t2/istio-proxy] {"bytes_sent":57,"downstream_remote_address":"100.64.110.3:0","upstream_local_address":"10.32.25.195:38886","protocol":"HTTP/2","request_id":"8f71f602-d2ac-93e5-b27e-78bdd557bbb4","response_code_details":"via_upstream","method":"GET","authorization":null,"upstream_transport_failure_reason":null,"upstream_cluster":"outbound|80||httpbin.org","response_flags":"-","grpc_status":"Unknown","duration":269,"response_code":200,"bytes_received":0,"connection_termination_details":null,"x_forwarded_for":"100.64.110.3,100.64.110.3","requested_server_name":null,"start_time":"2022-07-05T16:27:50.351Z","upstream_service_time":"269","route_name":null,"downstream_local_address":"10.32.25.195:8080","upstream_host":"3.94.154.124:443","path":"/ip","authority":"httpbin.org","user_agent":"curl/7.79.1"}