Skip to content

Runtime error when creating two layer 7 network policies #39535

New issue
New issue
Closed
Closed
Runtime error when creating two layer 7 network policies#39535
Assignees
fristonio
Labels
area/agentCilium agent related.area/proxyImpacts proxy components, including DNS, Kafka, Envoy and/or XDS servers.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.sig/policyImpacts whether traffic is allowed or denied based on user-defined policies.
@flocknroll

Description

@flocknroll
flocknroll
opened on May 14, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Version

equal or higher than v1.17.3 and lower than v1.18.0

What happened?

We're running Cilium in an AWS EKS cluster.
AMI is Bottlerocket.

Creating two layer 7 network policies (with TLS inspection) on the same namespace triggers a runtime error on the impacted nodes.

For example we created:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  namespace: namespace
spec:
  egress:
  - toPorts:
    - originatingTLS:
        secret:
          name: tls-certificates-orig
          namespace: cilium
      ports:
      - port: "443"
        protocol: TCP
      rules:
        http:
        - host: cloud.mongodb.com
          method: GET
          path: /api/atlas/v2/.*
      terminatingTLS:
        secret:
          name: tls-certificate-cloud.mongodb.com
          namespace: cilium
  - toPorts:
    - originatingTLS:
        secret:
          name: tls-certificates-orig
          namespace: cilium
      ports:
      - port: "443"
        protocol: TCP
      rules:
        http:
        - host: cbonds.com
          method: GET
          path: /api/.*
      terminatingTLS:
        secret:
          name: tls-certificate-cbonds.com
          namespace: cilium
  endpointSelector: {}

The network on the nodes hosting pods on namespace becomes unresponsive.
We can't even do a DNS lookup, we get timeouts.

When trying to restart the ciliumagent, we get the following error (showing only the last lines):

cilium-agent time="2025-05-14T14:03:12.742772276Z" level=info msg="Launching Cilium health daemon" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.74441635Z" level=info msg="Launching Cilium health endpoint" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.746092387Z" level=info msg="Deleted orphan backends" orphanBackends=0 subsys=service
cilium-agent time="2025-05-14T14:03:12.746252326Z" level=info msg="Starting initial GC of connection tracking" subsys=ct-nat-map-gc
cilium-agent time="2025-05-14T14:03:12.747127636Z" level=info msg="Cleaning up Cilium health endpoint" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.773138034Z" level=info msg="Started healthz status API server" address="127.0.0.1:9879" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.776210051Z" level=info msg="Daemon initialization completed" bootstrapTime=1.128353332s subsys=daemon
cilium-agent time="2025-05-14T14:03:12.776684679Z" level=info msg="Processing 0 queued deletion requests" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.77670554Z" level=info msg="Envoy: Waiting for endpoint restoration before serving xDS resources..." subsys=envoy-manager
cilium-agent time="2025-05-14T14:03:12.79358861Z" level=info msg="Conntrack garbage collector interval recalculated" actualPrevInterval=0s adjustedDeleteRatio=0.0098876953125 deleteRatio=0.0098876953125 expectedPrevInterval=5m0s newInterval=7m30s subsys=map-ct
cilium-agent time="2025-05-14T14:03:12.793656904Z" level=info msg="initial gc of ct and nat maps completed" duration=47.406278ms subsys=ct-nat-map-gc
cilium-agent time="2025-05-14T14:03:12.802008915Z" level=info msg="Initial scan of connection tracking completed, starting ctmap pressure metrics controller" subsys=ct-nat-map-gc
cilium-agent panic: runtime error: invalid memory address or nil pointer dereference
cilium-agent [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x2d7915c]
cilium-agent 
cilium-agent goroutine 1285 [running]:
cilium-agent github.com/cilium/cilium/pkg/policy.(*selectorPolicy).RedirectFilters.func1(0xc001cfc380)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/policy/resolve.go:362 +0x1c
cilium-agent github.com/cilium/cilium/pkg/endpoint.(*Endpoint).addNewRedirects(0xc0022e2008, {0x55baec8, 0x0}, 0x0)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:277 +0x24f
cilium-agent github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regeneratePolicy(0xc0022e2008, 0xc003d0cc18, 0xc003ad4580)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:247 +0x418
cilium-agent github.com/cilium/cilium/pkg/endpoint.(*Endpoint).ComputeInitialPolicy(0xc0022e2008, 0xc003d0cc08)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:828 +0xf8
cilium-agent github.com/cilium/cilium/pkg/endpoint.(*EndpointRegenerationEvent).Handle(0xc000cc60c0, 0xc001f62bd0)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/endpoint/events.go:39 +0x72
cilium-agent github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run.func1()
cilium-agent     /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:244 +0x131
cilium-agent sync.(*Once).doSlow(0xc002e82a10?, 0x4f38b5d?)
cilium-agent     /usr/local/go/src/sync/once.go:76 +0xb4
cilium-agent sync.(*Once).Do(...)
cilium-agent     /usr/local/go/src/sync/once.go:67
cilium-agent github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run(0x0?)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:232 +0x36
cilium-agent created by github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).Run in goroutine 1124
cilium-agent     /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:228 +0x69
stream closed EOF for cilium/cilium-j72j5 (cilium-agent)

Note
We actually wanted to restrict the L7 network policies to specific domains by adding:

- toPorts:
   ...
   toFQDNs:
     - matchName: cloud.mongodb.com

We stumbled upon the previous issue when we forgot to put the toFQDNs attributes.

How can we reproduce the issue?

  1. Install Cilium on a AWS EKS cluster with Helm
  2. values.yaml:
---
cni:
  install: true
  exclusive: true

ipam:
  mode: "cluster-pool"
  operator:
    clusterPoolIPv4PodCIDRList: ["10.244.0.0/16"]
policyEnforcementMode: "default"

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
          - key: eks.amazonaws.com/compute-type
            operator: NotIn
            values: ["fargate"]

ingressController:
  enabled: false

tls:
  secretsBackend: k8s
  secretSync:
    enabled: true

envoy:
  enabled: false

operator:
  replicas: 1
  unmanagedPodWatcher:
    restart: true

nodePort:
  enabled: false
hostPort:
  enabled: false

rollOutCiliumPods: true

certgen:
  generateCA: false
  1. Create a Cilium SSL private key and a self signed root certificate, trust it at the node level
  2. Create 2 private keys for the cloud.mongodb.com and cbonds.com TLS inspection
  3. Create the associated certificates and sign them with the Cilium root CA private key
  4. Store the PKs and certificates in the tls-certificate-cloud.mongodb.com and tls-certificate-cbonds.com secrets in the cilium namespace
  5. Put publicly trusted certificates in the tls-certificates-orig secret
  6. Create the CiliumNetworkPolicy object with the previous definition
  7. Try to access the network, e.g. curl https://www.github.com

Cilium version

1.17.3

Kernel Version

6.1.134 x86_64 GNU/Linux

Distribution is Bottlerocket: https://github.com/bottlerocket-os/bottlerocket

Kubernetes Version

AWS EKS 1.32

Regression

No response

Sysdump

No response

Relevant log output

apply-sysctl-overwrites unable to create cilium sysctl overwrites config: open /etc/sysctl.d/99-zzz-override_cilium.conf: permission denied
apply-sysctl-overwrites 
config Running
mount-cgroup time="2025-05-14T13:52:23.65609101Z" level=info msg="Mounted cgroupv2 filesystem at /run/cilium/cgroupv2" subsys=cgroups
mount-bpf-fs bpf on /sys/fs/bpf type bpf (rw,relatime,mode=700)
config 2025/05/14 13:52:22 INFO Starting hive
config time="2025-05-14T13:52:22.80266947Z" level=info msg="Establishing connection to apiserver" host="https://172.20.0.1:443" subsys=k8s-client
stream closed EOF for cilium/cilium-j72j5 (clean-cilium-state)
stream closed EOF for cilium/cilium-j72j5 (apply-sysctl-overwrites)
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-risk/transit-vpc-access-team-investment-risk-layer-7 policyRevision=31 deletedRules=0 identity="[24934 28645 47351]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-risk/vpce-access-team-investment-risk policyRevision=32 deletedRules=0 identity="[24934 28645 47351]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/access-own-team-investment-treasury policyRevision=33 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/ai-services-vpce-access-team-investment-treasury policyRevision=34 deletedRules=0 identity=[]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/all-ingress-access-team-investment-treasury policyRevision=35 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/cloud-accounts-access-team-investment-treasury policyRevision=36 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/coder-access-team-investment-treasury policyRevision=37 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/gitlab-access-team-investment-treasury policyRevision=38 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/intercept-all-dns policyRevision=39 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/kube-api-access-team-investment-treasury policyRevision=40 deletedRules=0 identity=[40869]
install-cni-binaries Installing cilium-cni to /host/opt/cni/bin/cilium-cni ...
install-cni-binaries Wrote /host/opt/cni/bin/cilium-cni
config time="2025-05-14T13:52:22.827869219Z" level=info msg="Connected to apiserver" subsys=k8s-client
config time="2025-05-14T13:52:22.829067837Z" level=info msg="Reading configuration from config-map:cilium/cilium-config" configSource="config-map:cilium/cilium-config" subsys=option-resolver
config time="2025-05-14T13:52:22.833974786Z" level=info msg="Got 151 config pairs from source" configSource="config-map:cilium/cilium-config" subsys=option-resolver
config time="2025-05-14T13:52:22.834027339Z" level=info msg="Reading configuration from cilium-node-config:cilium/" configSource="cilium-node-config:cilium/" subsys=option-resolver
config W0514 13:52:22.842186       1 warnings.go:70] cilium.io/v2alpha1 CiliumNodeConfig will be deprecated in cilium v1.16; use cilium.io/v2 CiliumNodeConfig
config time="2025-05-14T13:52:22.84228668Z" level=info msg="Got 0 config pairs from source" configSource="cilium-node-config:cilium/" subsys=option-resolver
config 2025/05/14 13:52:22 INFO Started duration=51.106656ms
config 2025/05/14 13:52:22 INFO Stopping
stream closed EOF for cilium/cilium-j72j5 (install-cni-binaries)
stream closed EOF for cilium/cilium-j72j5 (mount-bpf-fs)
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/nlb-access-team-investment-treasury policyRevision=41 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/on-premise-access-team-investment-treasury policyRevision=42 deletedRules=0 identity=[40869]
config 2025/05/14 13:52:22 INFO health.job-module-status-metrics (rev=2) module=health
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/snowflake-access-team-investment-treasury policyRevision=43 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/transit-vpc-access-team-investment-treasury policyRevision=44 deletedRules=0 identity=[40869]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-investment-treasury/vpce-access-team-investment-treasury policyRevision=45 deletedRules=0 identity=[40869]
stream closed EOF for cilium/cilium-j72j5 (mount-cgroup)
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/access-own-team-pam-tech-cts-cos policyRevision=46 deletedRules=0 identity="[10761 49198 51910 52712 59783 859 5061]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/ai-services-vpce-access-team-pam-tech-cts-cos policyRevision=47 deletedRules=0 identity=[51910]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/all-ingress-access-team-pam-tech-cts-cos policyRevision=48 deletedRules=0 identity="[10761 49198 51910 52712 59783 859 5061]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/cloud-accounts-access-team-pam-tech-cts-cos policyRevision=49 deletedRules=0 identity="[859 5061 10761 49198 51910 52712 59783]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/coder-access-team-pam-tech-cts-cos policyRevision=50 deletedRules=0 identity="[59783 859 5061 10761 49198 51910 52712]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/gitlab-access-team-pam-tech-cts-cos policyRevision=51 deletedRules=0 identity="[859 5061 10761 49198 51910 52712 59783]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/intercept-all-dns policyRevision=52 deletedRules=0 identity="[52712 59783 859 5061 10761 49198 51910]"
stream closed EOF for cilium/cilium-j72j5 (config)
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/kube-api-access-team-pam-tech-cts-cos policyRevision=53 deletedRules=0 identity="[52712 59783 859 5061 10761 49198 51910]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/nlb-access-team-pam-tech-cts-cos policyRevision=54 deletedRules=0 identity="[49198 51910 52712 59783 859 5061 10761]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/on-premise-access-team-pam-tech-cts-cos policyRevision=55 deletedRules=0 identity="[59783 859 5061 10761 49198 51910 52712]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/prefect-access-team-pam-tech-cts-cos policyRevision=56 deletedRules=0 identity="[10761 49198 51910 52712 59783 859 5061]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/snowflake-access-team-pam-tech-cts-cos policyRevision=57 deletedRules=0 identity="[51910 52712 59783 859 5061 10761 49198]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/transit-vpc-access-team-pam-tech-cts-cos policyRevision=58 deletedRules=0 identity="[59783 859 5061 10761 49198 51910 52712]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/transit-vpc-access-team-pam-tech-cts-cos-layer-7 policyRevision=59 deletedRules=0 identity="[49198 51910 52712 59783 859 5061 10761]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-cts-cos/vpce-access-team-pam-tech-cts-cos policyRevision=60 deletedRules=0 identity="[10761 49198 51910 52712 59783 859 5061]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/access-own-team-pam-tech-dat-ant policyRevision=61 deletedRules=0 identity="[7937 8513 11967 42148]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/ai-services-vpce-access-team-pam-tech-dat-ant policyRevision=62 deletedRules=0 identity=[]
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/all-ingress-access-team-pam-tech-dat-ant policyRevision=63 deletedRules=0 identity="[7937 8513 11967 42148]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/cloud-accounts-access-team-pam-tech-dat-ant policyRevision=64 deletedRules=0 identity="[7937 8513 11967 42148]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/coder-access-team-pam-tech-dat-ant policyRevision=65 deletedRules=0 identity="[42148 7937 8513 11967]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/gitlab-access-team-pam-tech-dat-ant policyRevision=66 deletedRules=0 identity="[8513 11967 42148 7937]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/intercept-all-dns policyRevision=67 deletedRules=0 identity="[8513 11967 42148 7937]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/kube-api-access-team-pam-tech-dat-ant policyRevision=68 deletedRules=0 identity="[7937 8513 11967 42148]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/nlb-access-team-pam-tech-dat-ant policyRevision=69 deletedRules=0 identity="[7937 8513 11967 42148]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/on-premise-access-team-pam-tech-dat-ant policyRevision=70 deletedRules=0 identity="[7937 8513 11967 42148]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Policy repository updates complete, triggering endpoint updates" module=agent.controlplane.policy policyRevision=70
cilium-agent time="2025-05-14T14:03:12.586075916Z" level=info msg="Imported CiliumNetworkPolicy" ciliumNetworkPolicyName=transit-vpc-access-team-pam-tech-dat-ant k8sApiVersion=cilium.io/v2 k8sNamespace=team-pam-tech-dat-ant subsys=policy-k8s-watcher
cilium-agent time="2025-05-14T14:03:12.586368513Z" level=info msg="Imported CiliumNetworkPolicy" ciliumNetworkPolicyName=vpce-access-team-pam-tech-dat-ant k8sApiVersion=cilium.io/v2 k8sNamespace=team-pam-tech-dat-ant subsys=policy-k8s-watcher
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Processing policy updates" module=agent.controlplane.policy count=4
cilium-agent time=2025-05-14T14:03:12Z level=info msg="inserting ipcache metadata for CIDR prefixes from policy" module=agent.controlplane.policy count=9
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/prefect-access-team-pam-tech-dat-ant policyRevision=71 deletedRules=0 identity="[42148 7937 8513 11967]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/snowflake-access-team-pam-tech-dat-ant policyRevision=72 deletedRules=0 identity="[11967 42148 7937 8513]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/transit-vpc-access-team-pam-tech-dat-ant policyRevision=73 deletedRules=0 identity="[7937 8513 11967 42148]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Upserted policy to repository" module=agent.controlplane.policy resource=cnp/team-pam-tech-dat-ant/vpce-access-team-pam-tech-dat-ant policyRevision=74 deletedRules=0 identity="[11967 42148 7937 8513]"
cilium-agent time=2025-05-14T14:03:12Z level=info msg="Policy repository updates complete, triggering endpoint updates" module=agent.controlplane.policy policyRevision=74
cilium-agent time="2025-05-14T14:03:12.633229381Z" level=info msg="All policy resources synchronized!" subsys=policy-k8s-watcher
cilium-agent time="2025-05-14T14:03:12.735456545Z" level=info msg="Regenerating restored endpoints" numRestored=4 subsys=daemon
cilium-agent time="2025-05-14T14:03:12.735803015Z" level=info msg="New endpoint" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1485 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
cilium-agent time="2025-05-14T14:03:12.73588383Z" level=info msg="New endpoint" ciliumEndpointName=kube-system/ebs-csi-node-qgbzk containerID=29f9188c6d containerInterface=eth0 datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1140 identity=1890 ipv4=10.244.36.2 ipv6= k8sPodName=kube-system/ebs-csi-node-qgbzk subsys=endpoint
cilium-agent time="2025-05-14T14:03:12.735923022Z" level=info msg="New endpoint" ciliumEndpointName=team-pam-tech-cts-cos/coder-team-pam-tech-cts-cos-fdosso-test-fdo-zsh-5f64bc949fkfgxh containerID=55dd8d5762 containerInterface=eth0 datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3888 identity=51910 ipv4=10.244.36.70 ipv6= k8sPodName=team-pam-tech-cts-cos/coder-team-pam-tech-cts-cos-fdosso-test-fdo-zsh-5f64bc949fkfgxh subsys=endpoint
cilium-agent time="2025-05-14T14:03:12.735974285Z" level=info msg="New endpoint" ciliumEndpointName=monitoring/fluent-bit-vxqtj containerID=ac3e0ef78d containerInterface=eth0 datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1596 identity=57863 ipv4=10.244.36.219 ipv6= k8sPodName=monitoring/fluent-bit-vxqtj subsys=endpoint
cilium-agent time="2025-05-14T14:03:12.736001697Z" level=info msg="Successfully restored endpoint. Scheduling regeneration" endpointID=1596 subsys=daemon
cilium-agent time="2025-05-14T14:03:12.736010377Z" level=info msg="Successfully restored endpoint. Scheduling regeneration" endpointID=3888 subsys=daemon
cilium-agent time="2025-05-14T14:03:12.736017528Z" level=info msg="Successfully restored endpoint. Scheduling regeneration" endpointID=1140 subsys=daemon
cilium-agent time="2025-05-14T14:03:12.736025288Z" level=info msg="Successfully restored endpoint. Scheduling regeneration" endpointID=1485 subsys=daemon
cilium-agent time="2025-05-14T14:03:12.736044729Z" level=info msg="Starting connection tracking garbage collector" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.737652292Z" level=info msg="Reusing existing global key" key="k8s:app.kubernetes.io/instance=fluent-bit;k8s:app.kubernetes.io/name=fluent-bit;k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=monitoring;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=fluentbit-service-account;k8s:io.kubernetes.pod.namespace=monitoring;" subsys=allocator
cilium-agent time="2025-05-14T14:03:12.73865905Z" level=info msg="Reusing existing global key" key="k8s:app.kubernetes.io/instance=coder-workspace-fdosso-test-fdo-zsh;k8s:app.kubernetes.io/name=coder-workspace;k8s:app.kubernetes.io/part-of=coder;k8s:com.coder.resource=true;k8s:com.coder.user.id=4ae094de-fbf3-4d5f-bb25-aa3199529e11;k8s:com.coder.user.username=fdosso;k8s:com.coder.username=fdosso;k8s:com.coder.workspace.base=python;k8s:com.coder.workspace.id=38b96284-05d9-4026-bf73-34ee497d3381;k8s:com.coder.workspace.instance.type=c5a.large;k8s:com.coder.workspace.name=test-fdo-zsh;k8s:com.coder.workspace.python=3.12;k8s:com.coder.workspace.tag=stable;k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=team-pam-tech-cts-cos;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=coder-workspace;k8s:io.kubernetes.pod.namespace=team-pam-tech-cts-cos;" subsys=allocator
cilium-agent time="2025-05-14T14:03:12.739071453Z" level=info msg="Reusing existing global key" key="k8s:app=ebs-csi-node;k8s:app.kubernetes.io/component=csi-driver;k8s:app.kubernetes.io/managed-by=EKS;k8s:app.kubernetes.io/name=aws-ebs-csi-driver;k8s:app.kubernetes.io/version=1.38.1;k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=ebs-csi-node-sa;k8s:io.kubernetes.pod.namespace=kube-system;" subsys=allocator
cilium-agent time="2025-05-14T14:03:12.742772276Z" level=info msg="Launching Cilium health daemon" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.74441635Z" level=info msg="Launching Cilium health endpoint" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.746092387Z" level=info msg="Deleted orphan backends" orphanBackends=0 subsys=service
cilium-agent time="2025-05-14T14:03:12.746252326Z" level=info msg="Starting initial GC of connection tracking" subsys=ct-nat-map-gc
cilium-agent time="2025-05-14T14:03:12.747127636Z" level=info msg="Cleaning up Cilium health endpoint" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.773138034Z" level=info msg="Started healthz status API server" address="127.0.0.1:9879" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.776210051Z" level=info msg="Daemon initialization completed" bootstrapTime=1.128353332s subsys=daemon
cilium-agent time="2025-05-14T14:03:12.776684679Z" level=info msg="Processing 0 queued deletion requests" subsys=daemon
cilium-agent time="2025-05-14T14:03:12.77670554Z" level=info msg="Envoy: Waiting for endpoint restoration before serving xDS resources..." subsys=envoy-manager
cilium-agent time="2025-05-14T14:03:12.79358861Z" level=info msg="Conntrack garbage collector interval recalculated" actualPrevInterval=0s adjustedDeleteRatio=0.0098876953125 deleteRatio=0.0098876953125 expectedPrevInterval=5m0s newInterval=7m30s subsys=map-ct
cilium-agent time="2025-05-14T14:03:12.793656904Z" level=info msg="initial gc of ct and nat maps completed" duration=47.406278ms subsys=ct-nat-map-gc
cilium-agent time="2025-05-14T14:03:12.802008915Z" level=info msg="Initial scan of connection tracking completed, starting ctmap pressure metrics controller" subsys=ct-nat-map-gc
cilium-agent panic: runtime error: invalid memory address or nil pointer dereference
cilium-agent [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x2d7915c]
cilium-agent 
cilium-agent goroutine 1285 [running]:
cilium-agent github.com/cilium/cilium/pkg/policy.(*selectorPolicy).RedirectFilters.func1(0xc001cfc380)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/policy/resolve.go:362 +0x1c
cilium-agent github.com/cilium/cilium/pkg/endpoint.(*Endpoint).addNewRedirects(0xc0022e2008, {0x55baec8, 0x0}, 0x0)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/endpoint/bpf.go:277 +0x24f
cilium-agent github.com/cilium/cilium/pkg/endpoint.(*Endpoint).regeneratePolicy(0xc0022e2008, 0xc003d0cc18, 0xc003ad4580)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:247 +0x418
cilium-agent github.com/cilium/cilium/pkg/endpoint.(*Endpoint).ComputeInitialPolicy(0xc0022e2008, 0xc003d0cc08)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/endpoint/policy.go:828 +0xf8
cilium-agent github.com/cilium/cilium/pkg/endpoint.(*EndpointRegenerationEvent).Handle(0xc000cc60c0, 0xc001f62bd0)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/endpoint/events.go:39 +0x72
cilium-agent github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run.func1()
cilium-agent     /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:244 +0x131
cilium-agent sync.(*Once).doSlow(0xc002e82a10?, 0x4f38b5d?)
cilium-agent     /usr/local/go/src/sync/once.go:76 +0xb4
cilium-agent sync.(*Once).Do(...)
cilium-agent     /usr/local/go/src/sync/once.go:67
cilium-agent github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).run(0x0?)
cilium-agent     /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:232 +0x36
cilium-agent created by github.com/cilium/cilium/pkg/eventqueue.(*EventQueue).Run in goroutine 1124
cilium-agent     /go/src/github.com/cilium/cilium/pkg/eventqueue/eventqueue.go:228 +0x69
stream closed EOF for cilium/cilium-j72j5 (cilium-agent)

Anything else?

No response

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

Labels

area/agentCilium agent related.area/proxyImpacts proxy components, including DNS, Kafka, Envoy and/or XDS servers.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.sig/policyImpacts whether traffic is allowed or denied based on user-defined policies.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions