Can't deploy bookinfo demo: ran out of forwarding rules limit #217
Description
I've completed deploying Istio with instructions here : https://istio.io/docs/tasks/installing-istio.html
Now I'm trying to deploy the BookInfo app: https://istio.io/docs/samples/bookinfo.html
I executed the following on an empty GKE cluster (just grafana stuff and istio installed):
$ kubectl apply -f <(istioctl kube-inject -f bookinfo.yaml)
service "details" created
deployment "details-v1" created
service "ratings" created
deployment "ratings-v1" created
service "reviews" created
deployment "reviews-v1" created
deployment "reviews-v2" created
deployment "reviews-v3" created
service "productpage" created
deployment "productpage-v1" created
ingress "gateway" created
Now I have the following services:
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
details 10.11.248.17 <none> 9080/TCP 13m
grafana 10.11.249.61 104.198.216.16 3000:32365/TCP 15m
istio-egress 10.11.246.0 <none> 80/TCP 15m
istio-ingress 10.11.240.31 104.197.201.14 80:31456/TCP 15m
istio-manager 10.11.242.117 <none> 8080/TCP,8081/TCP 15m
istio-mixer 10.11.244.141 <none> 9091/TCP,9094/TCP,42422/TCP 15m
kubernetes 10.11.240.1 <none> 443/TCP 30m
productpage 10.11.245.181 <none> 9080/TCP 13m
prometheus 10.11.246.122 <pending> 9090:31569/TCP 14m
ratings 10.11.249.96 <none> 9080/TCP 13m
reviews 10.11.254.160 <none> 9080/TCP 13m
servicegraph 10.11.255.233 <pending> 8088:31123/TCP 14m
as you can see some of them are <pending> on external IP. When I did describe on the service I saw
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
12m 1m 4 service-controller Normal UpdatedLoadBalancer Updated load balancer with new hosts
13m 45s 8 service-controller Normal CreatingLoadBalancer Creating load balancer
13m 24s 8 service-controller Warning CreatingLoadBalancerFailed Error creating load balancer (will retry): Failed to create load balancer for service default/servicegraph: failed to create forwarding rule a1a665b1534f911e7ae1242010a800fd: googleapi: Error 403: Quota 'FORWARDING_RULES' exceeded. Limit: 15.0, quotaExceeded
However my ingress has an IP address:
$ kd ing
Name: gateway
Namespace: default
Address: 104.197.201.14
Default backend: default-http-backend:80 (10.8.0.8:8080)
Rules:
Host Path Backends
---- ---- --------
*
/productpage productpage:9080 (<none>)
/login productpage:9080 (<none>)
/logout productpage:9080 (<none>)
Annotations:
Events: <none>
Note that the ingress doesn't have any GCLB url forwarding rules and when I go to https://pantheon.corp.google.com/networking/loadbalancing/loadBalancers/list I don't see the LB created for the ingress. The external IP exists but it doesn't do anything.
I realized I had to clean up some forwarding rules (screenshot below) from my previous clusters to open up some space. (Now I'm waiting for things to get fixed...)
I am aware that this is a GCP problem and not istio, but is there anything the docs can do to remediate this?