-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Closed
Labels
area/servicemeshGH issues or PRs regarding servicemeshGH issues or PRs regarding servicemeshkind/bugThis is a bug in the Cilium logic.This is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.This was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.This issue requires triaging to establish severity and next steps.
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
With cilium 1.14.3, set the default ingress class to shared. Create a basic ClusterIssuer for the Let's Encrypt ACME HTTP01 challenge. Create an Ingress with the appropriate annotation and spec.tls setup properly.
Cilium Install
export CILIUM_VERSION="v1.14.3"
helm upgrade --install \
cilium \
cilium/cilium \
--version ${CILIUM_VERSION} \
--namespace kube-system \
--set ipam.mode=kubernetes \
--set kubeProxyReplacement=true \
--set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
--set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
--set bpf.autoMount.enabled=true \
--set cgroup.autoMount.enabled=false \
--set cgroup.hostRoot=/sys/fs/cgroup \
--set k8sServiceHost=localhost \
--set k8sServicePort=7445 \
--set bpf.masquerade=true \
--set bandwidthManager.enabled=true \
--set bandwidthManager.bbr=true \
--set prometheus.enabled=true \
--set operator.prometheus.enabled=true \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set hubble.enabled=true \
--set hubble.metrics.enableOpenMetrics=true \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}" \
--set ingressController.enabled=true \
--set ingressController.default=true \
--set ingressController.loadbalancerMode=shared \
--set ingressController.service.allocateLoadBalancerNodePorts="true" \
--set ingressController.service.loadBalancerIP="10.0.0.153" \
--set l2announcements.enabled=true \
--set l2podAnnouncements.enabled=true \
--set l2podAnnouncements.interface=eth0 \
--set k8sClientRateLimit.qps=${QPS} \
--set k8sClientRateLimit.burst=${BURST} \
--set envoy.enabled=true \
--set envoy.prometheus.enabled=true \
--set routingMode=native \
--set ipv4NativeRoutingCIDR=10.0.0.0/8 \
--set policyEnforcementMode=default \
--set debug.enabled=false \
--set autoDirectNodeRoutes=true \
--set hostFirewall.enabled=true \
--set loadBalancer.algorithm=maglev \
--set maglev.tableSize=65521 \
--set maglev.hashSeed=${SEED} \
--set loadBalancer.mode=dsr \
--set loadBalancer.dsrDispatch=opt \
--values <(cat <<EOF
ingressController:
service:
labels:
cilium.loadbalancer.ips.service/name: ingress-gateway-pool
EOF
)
Cert-manager install
export CERT_MANAGER_VERSION="v1.13.1"
helm upgrade --install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true \
--version $CERT_MANAGER_VERSION \
--set "extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true,--feature-gates=AdditionalCertificateOutputFormats=true}" \
--set webhook.extraArgs={--feature-gates="AdditionalCertificateOutputFormats=true"}
ZeroSSL Cluster Issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: cilium-zerossl
spec:
acme:
# ZeroSSL ACME server
server: https://acme.zerossl.com/v2/DV90
email: ${CLUSTER_ISSUER_EMAIL}
privateKeySecretRef:
name: zerossl-private-key
externalAccountBinding:
keyID: ${ZERO_SSL_EAB_KEY_ID}
keySecretRef:
name: zerossl-eab-secret
key: secret
# ACME HTTP01 Ingress solver
solvers:
- http01:
ingress:
ingressClassName: cilium
LB Pool and Announcement Policy (No BGP)
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
name: ingress-gateway-pool
spec:
disabled: false
cidrs:
- cidr: 10.0.0.152/30
serviceSelector:
matchLabels:
cilium.loadbalancer.ips.service/name: ingress-gateway-pool
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
name: ingress-gateway-pool-announcement-policy
spec:
loadBalancerIPs: true
interfaces:
- eth0
serviceSelector:
matchLabels:
cilium.loadbalancer.ips.service/name: ingress-gateway-pool
Ingress Service
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ${INGRESS_NAME}
namespace: ${NAMESPACE}
annotations:
cert-manager.io/cluster-issuer: cilium-zerossl
spec:
ingressClassName: cilium
rules:
- host: ${SERVICE_HOST}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ${SERVICE_NAME}
port:
number: 3001
tls:
- hosts:
- ${SERVICE_HOST}
secretName: cilium-zerossl-${INGRESS_NAME}-tls
Observe as cert manager spins in a loop and creates hundreds of ingresses.
Cilium Version
cilium-cli: v0.15.11 compiled with go1.21.3 on linux/amd64
cilium image (default): v1.14.2
cilium image (stable): v1.14.3
cilium image (running): 1.14.3
Kernel Version
Linux: 6.1.58
Kubernetes Version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
Sysdump
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct
- I agree to follow this project's Code of Conduct
cornfeedhobo
Metadata
Metadata
Assignees
Labels
area/servicemeshGH issues or PRs regarding servicemeshGH issues or PRs regarding servicemeshkind/bugThis is a bug in the Cilium logic.This is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.This was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.This issue requires triaging to establish severity and next steps.