Ingress Controller: CiliumEnvoyConfig deleted in shared load balancer mode after deleting Cilium's IngressClass #28691
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
I deployed Cilium with the Ingress Controller enabled into a kind cluster, using the following commands:
WORKERS=3 KUBEPROXY_MODE=none make kind
make kind-image
And by adding the following to contrib/testing/kind-values.yaml:
debug:
enabled: true
ingressController:
enabled: true
default: true
loadbalancerMode: shared
kubeProxyReplacement: strict
k8sServiceHost: 172.21.0.4
k8sServicePort: 6443
I created two dummy Ingresses:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress1
spec:
ingressClassName: cilium
rules:
- http:
paths:
- path: /1
pathType: Prefix
backend:
service:
name: service1
port:
number: 80
---
apiVersion: v1
kind: Service
metadata:
name: service1
spec:
selector:
app.kubernetes.io/name: app1
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
name: app1
labels:
app.kubernetes.io/name: app1
spec:
containers:
- name: nginx
image: nginx:stable
ports:
- containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress2
spec:
ingressClassName: cilium
rules:
- http:
paths:
- path: /2
pathType: Prefix
backend:
service:
name: service2
port:
number: 80
---
apiVersion: v1
kind: Service
metadata:
name: service2
spec:
selector:
app.kubernetes.io/name: app2
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
name: app2
labels:
app.kubernetes.io/name: app2
spec:
containers:
- name: nginx
image: nginx:stable
ports:
- containerPort: 80After creating these Ingress resources, I edited Ingress2 to remove the ingressClassName from the spec:
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress1 cilium * 80 15s
ingress2 <none> * 80 15s
Then I deleted the Cilium IngressClass from the cluster and noticed that the shared CiliumEnvoyConfig named cilium-ingress in kube-system was removed:
$ kubectl get ciliumenvoyconfig -A
NAMESPACE NAME AGE
kube-system cilium-ingress 25s
$ kubectl delete ingressclass cilium
ingressclass.networking.k8s.io "cilium" deleted
$ kubectl get ciliumenvoyconfig -A
No resources found
This is unexpected, because the shared CiliumEnvoyConfig should still be present to handle ingress1, which specifically names Cilium as the ingress to use.
Cilium Version
Deployed from the cilium repo with tag 1.14.2 checked out.
Kernel Version
Linux houston 6.1.55 #1-NixOS SMP PREEMPT_DYNAMIC Sat Sep 23 09:11:13 UTC 2023 x86_64 GNU/Linux
Kubernetes Version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"archive", BuildDate:"1980-01-01T00:00:00Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-30T06:34:50Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Sysdump
No response
Relevant log output
level=debug msg="Handling ingress class deleted event" ingressClass=cilium subsys=ingress-controller
level=debug msg="Handling ingress class delete" ingressClass="&IngressClass{ObjectMeta:{cilium cf8df0df-a134-4d14-b6dd-88a6d0c32b21 8562 1 <nil> map[app.kubernetes.io/managed-by:Helm] map[ingressclass.kubernetes.io/is-default-class:true meta.helm.sh/release-name:cilium meta.helm.sh/release-namespace:kube-system] []},Spec:IngressClassSpec{Controller:cilium.io/ingress-controller,Parameters:nil,},}" subsys=ingress-controller
level=debug msg="Handling cilium ingress class deleted event" ingressClass=cilium subsys=ingress-controller
level=debug msg="Cilium IngressClass deleted" subsys=ingress-controller
level=debug msg="Generated model for ingress" forcedShared=false ingress=ingress2 k8sNamespace=default model="&{[{ing-ingress1-default-* [{ingress1 default v1 Ingress 1b1bf11d-6255-4765-b846-2e3fbeef1d95}] 80 * [] [{ [] prefix:/1 [] [] <nil> [{service1 default 0xc001260e28 <nil>}] <nil> <nil> <nil> <nil> <nil> <nil>}] <nil>} {ing-ingress2-default-* [{ingress2 default v1 Ingress e0c96217-d6c4-43bc-96dc-ff0109ac9b87}] 80 * [] [{ [] prefix:/2 [] [] <nil> [{service2 default 0xc001260e58 <nil>}] <nil> <nil> <nil> <nil> <nil> <nil>}] <nil>}] []}" subsys=ingress-controller
level=debug msg="Translated resources for ingress" ciliumEnvoyConfig="&{{ } {cilium-ingress kube-system 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [{v1 Ingress ingress1 1b1bf11d-6255-4765-b846-2e3fbeef1d95 <nil> <nil>} {v1 Ingress ingress2 e0c96217-d6c4-43bc-96dc-ff0109ac9b87 <nil> <nil>}] [] []} {[0xc000eabf20] [0xc000b19b40 0xc000b19b80] [[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:\"listener\" filter_chains:{filter_chain_match:{transport_protocol:\"raw_buffer\"} filters:{name:\"envoy.filters.network.http_connection_manager\" typed_config:{[type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager]:{stat_prefix:\"listener-insecure\" rds:{route_config_name:\"listener-insecure\"} http_filters:{name:\"envoy.filters.http.router\" typed_config:{[type.googleapis.com/envoy.extensions.filters.http.router.v3.Router]:{}}} use_remote_address:{value:true} upgrade_configs:{upgrade_type:\"websocket\"}}}}} listener_filters:{name:\"envoy.filters.listener.tls_inspector\" typed_config:{[type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector]:{}}} socket_options:{description:\"Enable TCP keep-alive (default to enabled)\" level:1 name:9 int_value:1 state:STATE_LISTENING} socket_options:{description:\"TCP keep-alive idle time (in seconds) (defaults to 10s)\" level:6 name:4 int_value:10 state:STATE_LISTENING} socket_options:{description:\"TCP keep-alive probe intervals (in seconds) (defaults to 5s)\" level:6 name:5 int_value:5 state:STATE_LISTENING} socket_options:{description:\"TCP keep-alive probe max failures.\" level:6 name:6 int_value:10 state:STATE_LISTENING}} [type.googleapis.com/envoy.config.route.v3.RouteConfiguration]:{name:\"listener-insecure\" virtual_hosts:{name:\"*\" domains:\"*\" routes:{match:{path_separated_prefix:\"/1\"} route:{cluster:\"default/service1:80\" max_stream_duration:{max_stream_duration:{}}}} routes:{match:{path_separated_prefix:\"/2\"} route:{cluster:\"default/service2:80\" max_stream_duration:{max_stream_duration:{}}}}}} [type.googleapis.com/envoy.config.cluster.v3.Cluster]:{name:\"default/service1:80\" type:EDS connect_timeout:{seconds:5} typed_extension_protocol_options:{key:\"envoy.extensions.upstreams.http.v3.HttpProtocolOptions\" value:{[type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions]:{common_http_protocol_options:{idle_timeout:{seconds:60}} use_downstream_protocol_config:{http2_protocol_options:{}}}}} outlier_detection:{split_external_local_origin_errors:true}} [type.googleapis.com/envoy.config.cluster.v3.Cluster]:{name:\"default/service2:80\" type:EDS connect_timeout:{seconds:5} typed_extension_protocol_options:{key:\"envoy.extensions.upstreams.http.v3.HttpProtocolOptions\" value:{[type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions]:{common_http_protocol_options:{idle_timeout:{seconds:60}} use_downstream_protocol_config:{http2_protocol_options:{}}}}} outlier_detection:{split_external_local_origin_errors:true}}]}}" endpoint=nil ingress=ingress2 k8sNamespace=default service=nil subsys=ingress-controller
level=debug msg="Deleted CiliumEnvoyConfig" ciliumEnvoyConfigName=cilium-ingress subsys=ingress-controllerAnything else?
Related: #28289
Code of Conduct
- I agree to follow this project's Code of Conduct