Skip to content

Enabling Host Firewall causes: BPF program is too large #38967

@kobajagi

Description

@kobajagi

Is there an existing issue for this?

  • I have searched the existing issues

Version

equal or higher than v1.17.2 and lower than v1.18.0

What happened?

When I enable host-firewall in existing cluster I get "BPF program is too large" error in cilium pods. I updated Cilium to the latest (1.17.3) but still get the error. Turning off host-firewall fixes issue.

Our cilium config is bellow. Are there any known config incompatibilities with host-firewall that needs to be tweaked to get BPF program size bellow limit?

How can we reproduce the issue?

Use cilium configmap:

$ sudo kubectl get cm cilium-config -n kube-system -o yaml

apiVersion: v1
data:
  agent-not-ready-taint-key: node.cilium.io/agent-not-ready
  allow-localhost: policy
  arping-refresh-period: 30s
  auto-direct-node-routes: "false"
  bgp-secrets-namespace: kube-system
  bpf-events-drop-enabled: "true"
  bpf-events-policy-verdict-enabled: "true"
  bpf-events-trace-enabled: "true"
  bpf-lb-acceleration: disabled
  bpf-lb-algorithm-annotation: "false"
  bpf-lb-external-clusterip: "false"
  bpf-lb-map-max: "65536"
  bpf-lb-mode-annotation: "false"
  bpf-lb-sock: "false"
  bpf-lb-source-range-all-types: "false"
  bpf-map-dynamic-size-ratio: "0.0025"
  bpf-policy-map-max: "16384"
  bpf-root: /sys/fs/bpf
  cgroup-root: /sys/fs/cgroup
  cilium-endpoint-gc-interval: 5m0s
  cluster-id: "0"
  cluster-name: kube-gva2-lab-3-management
  cluster-pool-ipv4-cidr: 172.20.104.0/22
  cluster-pool-ipv4-mask-size: "26"
  cluster-pool-ipv6-cidr: 2a04:c43:e00:147a:500:c::/97
  cluster-pool-ipv6-mask-size: "104"
  clustermesh-enable-endpoint-sync: "false"
  clustermesh-enable-mcs-api: "false"
  cni-exclusive: "true"
  cni-log-file: /var/run/cilium/cilium-cni.log
  controller-group-metrics: write-cni-file sync-host-ips sync-lb-maps-with-k8s-services
  custom-cni-conf: "false"
  datapath-mode: veth
  debug: "false"
  default-lb-service-ipam: lbipam
  direct-routing-skip-unreachable: "false"
  dnsproxy-enable-transparent-mode: "true"
  dnsproxy-socket-linger-timeout: "10"
  egress-gateway-reconciliation-trigger-interval: 1s
  enable-auto-protect-node-port-range: "true"
  enable-bgp-control-plane: "true"
  enable-bgp-control-plane-status-report: "true"
  enable-bpf-clock-probe: "false"
  enable-bpf-masquerade: "true"
  enable-bpf-tproxy: "true"
  enable-endpoint-health-checking: "true"
  enable-endpoint-lockdown-on-policy-overflow: "false"
  enable-experimental-lb: "false"
  enable-health-check-loadbalancer-ip: "false"
  enable-health-check-nodeport: "true"
  enable-health-checking: "true"
  enable-host-firewall: "true"
  enable-hubble: "true"
  enable-internal-traffic-policy: "true"
  enable-ipv4: "true"
  enable-ipv4-big-tcp: "false"
  enable-ipv4-masquerade: "true"
  enable-ipv6: "true"
  enable-ipv6-big-tcp: "false"
  enable-ipv6-masquerade: "true"
  enable-k8s-networkpolicy: "true"
  enable-k8s-terminating-endpoint: "true"
  enable-l2-neigh-discovery: "true"
  enable-l7-proxy: "true"
  enable-lb-ipam: "true"
  enable-local-redirect-policy: "false"
  enable-masquerade-to-route-source: "false"
  enable-metrics: "true"
  enable-node-selector-labels: "false"
  enable-non-default-deny-policies: "true"
  enable-pmtu-discovery: "true"
  enable-policy: always
  enable-policy-secrets-sync: "true"
  enable-runtime-device-detection: "true"
  enable-sctp: "false"
  enable-source-ip-verification: "true"
  enable-svc-source-range-check: "true"
  enable-tcx: "true"
  enable-vtep: "false"
  enable-well-known-identities: "false"
  enable-xt-socket-fallback: "true"
  envoy-access-log-buffer-size: "4096"
  envoy-base-id: "0"
  envoy-keep-cap-netbindservice: "false"
  external-envoy-proxy: "false"
  health-check-icmp-failure-threshold: "3"
  http-retry-count: "3"
  hubble-disable-tls: "false"
  hubble-export-file-max-backups: "5"
  hubble-export-file-max-size-mb: "10"
  hubble-listen-address: :4244
  hubble-socket-path: /var/run/cilium/hubble.sock
  hubble-tls-cert-file: /var/lib/cilium/tls/hubble/server.crt
  hubble-tls-client-ca-files: /var/lib/cilium/tls/hubble/client-ca.crt
  hubble-tls-key-file: /var/lib/cilium/tls/hubble/server.key
  identity-allocation-mode: crd
  identity-gc-interval: 15m0s
  identity-heartbeat-timeout: 30m0s
  install-no-conntrack-iptables-rules: "false"
  ipam: cluster-pool
  ipam-cilium-node-update-rate: 15s
  iptables-random-fully: "false"
  k8s-require-ipv4-pod-cidr: "false"
  k8s-require-ipv6-pod-cidr: "false"
  kube-proxy-replacement: "true"
  kube-proxy-replacement-healthz-bind-address: ""
  max-connected-clusters: "255"
  mesh-auth-enabled: "true"
  mesh-auth-gc-interval: 5m0s
  mesh-auth-queue-size: "1024"
  mesh-auth-rotated-identities-queue-size: "1024"
  monitor-aggregation: medium
  monitor-aggregation-flags: all
  monitor-aggregation-interval: 5s
  nat-map-stats-entries: "32"
  nat-map-stats-interval: 30s
  node-port-bind-protection: "true"
  nodes-gc-interval: 5m0s
  operator-api-serve-addr: 127.0.0.1:9234
  operator-prometheus-serve-addr: :9963
  policy-audit-mode: "false"
  policy-cidr-match-mode: ""
  policy-secrets-namespace: kube-system
  policy-secrets-only-from-secrets-namespace: "true"
  preallocate-bpf-maps: "false"
  procfs: /host/proc
  prometheus-serve-addr: :9962
  proxy-connect-timeout: "2"
  proxy-idle-timeout-seconds: "60"
  proxy-initial-fetch-timeout: "30"
  proxy-max-concurrent-retries: "128"
  proxy-max-connection-duration-seconds: "0"
  proxy-max-requests-per-connection: "0"
  proxy-prometheus-port: "9964"
  proxy-xff-num-trusted-hops-egress: "0"
  proxy-xff-num-trusted-hops-ingress: "0"
  remove-cilium-node-taints: "true"
  routing-mode: tunnel
  service-no-backend-response: reject
  set-cilium-is-up-condition: "true"
  set-cilium-node-taints: "true"
  synchronize-k8s-nodes: "true"
  tofqdns-dns-reject-response-code: refused
  tofqdns-enable-dns-compression: "true"
  tofqdns-endpoint-max-ip-per-hostname: "1000"
  tofqdns-idle-connection-grace-period: 0s
  tofqdns-max-deferred-connection-deletes: "10000"
  tofqdns-proxy-response-max-delay: 100ms
  tunnel-protocol: vxlan
  unmanaged-pod-watcher-interval: "15"
  vtep-cidr: ""
  vtep-endpoint: ""
  vtep-mac: ""
  vtep-mask: ""
  write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist

Cilium Version

1.17.3

Kernel Version

5.15.0-130-generic

Kubernetes Version

1.32.3

Regression

No response

Sysdump

No response

Relevant log output

level=error msg="Error while reloading endpoint BPF program" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPoli
cyRevision=1535 endpointID=3799 error="attaching cilium_host: loading eBPF collection into the kernel: program tail_nodeport_nat_ingress_ipv4: load program: argument list too long: BPF prog
ram is too large. Processed 1000001 insn (1259 line(s) omitted)" identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint

Anything else?

No response

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.kind/complexity-issueRelates to BPF complexity or program size issuesneeds/triageThis issue requires triaging to establish severity and next steps.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions