Skip to content

GatewayAPI: Incorrect HTTPRoutes traffic forwarding  #24318

New issue
New issue
Closed
Closed
GatewayAPI: Incorrect HTTPRoutes traffic forwarding #24318
Labels
area/agentCilium agent related.area/servicemeshGH issues or PRs regarding servicemeshfeature/k8s-gateway-apifeature/k8s-ingresskind/community-reportThis was reported by a user in the Cilium community, eg via Slack.kind/enhancementThis would improve or streamline existing functionality.
@log1cb0mb

Description

@log1cb0mb
log1cb0mb
opened on Mar 12, 2023

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Problem:
Running cilium in direct routing mode and there is external BGP ECMP/LB for LB type services. Ingress traffic is load balanced to cilium nodes. Httproutes traffic works fine when traffic ends up on any of the cilium-node proxy (proxy through node’s ingress IP) except from the same cilium-node where target backend pod/service is.

Behaviour (scenario where traffic is proxied through same cilium-node as backend service):

Proxy traffic from cilium-node to backend app pod is routed/forwarded internally however the return traffic from backend pod to cilium-node proxy/ingress IP is forwarded to host's default gateway.

As cilium proxy/ingress ip is allocated from the same pod CIDR block assigned to node, there are following routes e.g:

10.53.0.192/26 via 10.53.0.214 dev cilium_host src 10.53.0.214
10.53.0.214 dev cilium_host scope link

where 10.53.0.214 being the internal cilium router. And cilium ingress IP in this case was 10.53.0.217 so as expected with standard kernel routing, packet to be forward to cilium internal router as can be seen from routing entry above but no, based on traffic captures packets are forwarded to default gateway (DST MAC address).

This issue is resolved or works as expected (traffic is routed/forwarded internally) if enable-endpoint-routes option is enabled.

Expectation:
Traffic where source and destination IPs are reachable from node's local routing stack, it should not be forwarded to default gateway.

Local environment to reproduce the issue:
I have created a project k8s-kind-clab-cilium that can be used to spin up a local environment and reproduce the issue. Cilium config in the environment closely matches to my other environments mainly from routing topology perspective. The project includes deploying Gateway API example as well.

Notes:
FRR based routers being used in the project are bit flaky with ECMP as it randomly picks next hops for routing. In order to reproduce the issue or observe, ensure that upstream ToRs are forwarding traffic to same cilium-node as the schedule backend pod. If not done automatically then shutdown all BGP neighbours from ToRs except target node.

tcpdump can be installed on the target node to verify MAC address in traffic capture. cilium ingress/proxy IPs can be pulled before captures incase traffic is load balanced between cilium nodes and pinning local node ingress IP:

kubectl get ciliumnodes -o jsonpath="{range .items[*]}{'Ingress/Proxy IP: '}{.spec.ingress.ipv4}{'\t'}{'Node: '}{.metadata.name}{'\n'}{end}"

Sample capture:

# ip n (to match default gw MAC in captures)
# tcpdump -e -nni net0 tcp port 9080

This should show for e.g SYN-ACK packet from app pod IP to node's ingress IP with destination MAC of default gateway.

Cilium Version

v1.13.0

Kernel Version

Linux worker 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Same issue on bare metal with RHEL8 4.18.x

Kubernetes Version

v1.125.3

Sysdump

cilium-sysdump-20230312-190021.zip

Relevant log output

### Cilium monitor -j ###
{"cpu":"CPU 03:","type":"debug","message":"Successfully mapped addr=192.168.10.100 to identity=2"}
{"cpu":"CPU 03:","type":"debug","message":"CT entry found lifetime=49730, revnat=0"}
{"cpu":"CPU 03:","type":"debug","message":"CT created 1/2: sport=0 dport=49730 nexthdr=0 flags= revnat=0"}
{"cpu":"CPU 03:","type":"debug","message":"Successfully mapped addr=192.168.10.100 to identity=2"}
{"cpu":"CPU 03:","type":"debug","message":"CT entry found lifetime=49730, revnat=0"}
{"cpu":"CPU 03:","type":"debug","message":"CT created 1/2: sport=0 dport=49730 nexthdr=0 flags= revnat=0"}
{"cpu":"CPU 03:","type":"debug","message":"Successfully mapped addr=192.168.10.100 to identity=2"}
{"cpu":"CPU 03:","type":"debug","message":"CT entry found lifetime=49730, revnat=0"}
{"cpu":"CPU 03:","type":"debug","message":"CT created 1/2: sport=0 dport=49730 nexthdr=0 flags= revnat=0"}
{"cpu":"CPU 03:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}
{"cpu":"CPU 03:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}
{"cpu":"CPU 01:","type":"debug","message":"Inheriting identity=2 from stack"}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 01:","type":"debug","message":"Attempting local delivery for container id 3734 from seclabel 8"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.235:39775 dst=10.51.2.119:9080"}
{"cpu":"CPU 03:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT verdict: New, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack create: proxy-port=0 revnat=0 src-identity=8 lb=0.0.0.0"}
{"cpu":"CPU 01:","type":"trace","mark":"0x4037ec46","ifindex":"lxce564053d40dc","state":"new","observationPoint":"to-endpoint","traceSummary":"-\u003e endpoint 3734","source":3734,"bytes":74,"srcLabel":8,"dstLabel":46123,"dstID":3734,"summary":{"ethernet":"Ethernet\t{Contents=[..14..] Payload=[..62..] SrcMAC=52:98:da:91:c8:30 DstMAC=92:be:64:23:e3:e5 EthernetType=IPv4 Length=0}","ipv4":"IPv4\t{Contents=[..20..] Payload=[..40..] Version=4 IHL=5 TOS=0 Length=60 Id=49855 Flags=DF FragOffset=0 TTL=63 Protocol=TCP Checksum=24373 SrcIP=10.51.2.235 DstIP=10.51.2.119 Options=[] Padding=[]}","tcp":"TCP\t{Contents=[..40..] Payload=[] SrcPort=39775 DstPort=9080(glrpc) Seq=3072567746 Ack=0 DataOffset=10 FIN=false SYN=true RST=false PSH=false ACK=false URG=false ECE=false CWR=false NS=false Window=64240 Checksum=6646 Urgent=0 Options=[..5..] Padding=[]}","l2":{"src":"52:98:da:91:c8:30","dst":"92:be:64:23:e3:e5"},"l3":{"src":"10.51.2.235","dst":"10.51.2.119"},"l4":{"src":"39775","dst":"9080"}}}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.119:9080 dst=10.51.2.235:39775"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=1"}
{"cpu":"CPU 01:","type":"debug","message":"CT entry found lifetime=15518, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT verdict: Reply, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 01:","type":"trace","mark":"0xa8cd380a","ifindex":"net0","state":"reply","observationPoint":"to-network","traceSummary":"-\u003e network","source":3734,"bytes":74,"srcLabel":46123,"dstLabel":8,"dstID":0,"summary":{"ethernet":"Ethernet\t{Contents=[..14..] Payload=[..62..] SrcMAC=92:be:64:23:e3:e5 DstMAC=52:98:da:91:c8:30 EthernetType=IPv4 Length=0}","ipv4":"IPv4\t{Contents=[..20..] Payload=[..40..] Version=4 IHL=5 TOS=0 Length=60 Id=0 Flags=DF FragOffset=0 TTL=63 Protocol=TCP Checksum=8693 SrcIP=10.51.2.119 DstIP=10.51.2.235 Options=[] Padding=[]}","tcp":"TCP\t{Contents=[..40..] Payload=[] SrcPort=9080(glrpc) DstPort=39775 Seq=232423253 Ack=3072567747 DataOffset=10 FIN=false SYN=true RST=false PSH=false ACK=true URG=false ECE=false CWR=false NS=false Window=65160 Checksum=6646 Urgent=0 Options=[..5..] Padding=[]}","l2":{"src":"92:be:64:23:e3:e5","dst":"52:98:da:91:c8:30"},"l3":{"src":"10.51.2.119","dst":"10.51.2.235"},"l4":{"src":"9080","dst":"39775"}}}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=10.51.2.119 to identity=46123"}
{"cpu":"CPU 01:","type":"debug","message":"Inheriting identity=2 from stack"}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 01:","type":"debug","message":"Attempting local delivery for container id 3734 from seclabel 8"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.235:39775 dst=10.51.2.119:9080"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT entry found lifetime=15518, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT verdict: Established, revnat=0"}
{"cpu":"CPU 01:","type":"trace","mark":"0x4037ec46","ifindex":"lxce564053d40dc","state":"established","observationPoint":"to-endpoint","traceSummary":"-\u003e endpoint 3734","source":3734,"bytes":66,"srcLabel":8,"dstLabel":46123,"dstID":3734,"summary":{"ethernet":"Ethernet\t{Contents=[..14..] Payload=[..54..] SrcMAC=52:98:da:91:c8:30 DstMAC=92:be:64:23:e3:e5 EthernetType=IPv4 Length=0}","ipv4":"IPv4\t{Contents=[..20..] Payload=[..32..] Version=4 IHL=5 TOS=0 Length=52 Id=49856 Flags=DF FragOffset=0 TTL=63 Protocol=TCP Checksum=24380 SrcIP=10.51.2.235 DstIP=10.51.2.119 Options=[] Padding=[]}","tcp":"TCP\t{Contents=[..32..] Payload=[] SrcPort=39775 DstPort=9080(glrpc) Seq=3072567747 Ack=232423254 DataOffset=8 FIN=false SYN=false RST=false PSH=false ACK=true URG=false ECE=false CWR=false NS=false Window=502 Checksum=6638 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:208856449/2273117105 0x0c72e581877d03b1)] Padding=[]}","l2":{"src":"52:98:da:91:c8:30","dst":"92:be:64:23:e3:e5"},"l3":{"src":"10.51.2.235","dst":"10.51.2.119"},"l4":{"src":"39775","dst":"9080"}}}
{"cpu":"CPU 01:","type":"debug","message":"Inheriting identity=2 from stack"}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 01:","type":"debug","message":"Attempting local delivery for container id 3734 from seclabel 8"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.235:39775 dst=10.51.2.119:9080"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT entry found lifetime=37058, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT verdict: Established, revnat=0"}
{"cpu":"CPU 01:","type":"trace","mark":"0x4037ec46","ifindex":"lxce564053d40dc","state":"established","observationPoint":"to-endpoint","traceSummary":"-\u003e endpoint 3734","source":3734,"bytes":228,"srcLabel":8,"dstLabel":46123,"dstID":3734,"summary":{"ethernet":"Ethernet\t{Contents=[..14..] Payload=[..118..] SrcMAC=52:98:da:91:c8:30 DstMAC=92:be:64:23:e3:e5 EthernetType=IPv4 Length=0}","ipv4":"IPv4\t{Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=214 Id=49857 Flags=DF FragOffset=0 TTL=63 Protocol=TCP Checksum=24217 SrcIP=10.51.2.235 DstIP=10.51.2.119 Options=[] Padding=[]}","tcp":"TCP\t{Contents=[..32..] Payload=[..66..] SrcPort=39775 DstPort=9080(glrpc) Seq=3072567747 Ack=232423254 DataOffset=8 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=502 Checksum=6800 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:208856449/2273117105 0x0c72e581877d03b1)] Padding=[]}","l2":{"src":"52:98:da:91:c8:30","dst":"92:be:64:23:e3:e5"},"l3":{"src":"10.51.2.235","dst":"10.51.2.119"},"l4":{"src":"39775","dst":"9080"}}}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.119:9080 dst=10.51.2.235:39775"}
{"cpu":"CPU 01:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=1"}
{"cpu":"CPU 01:","type":"debug","message":"CT entry found lifetime=37058, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT verdict: Reply, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=10.51.2.119 to identity=46123"}
{"type":"logRecord","observationPoint":"Egress","flowType":"Request","l7Proto":"http","srcEpID":0,"srcEpLabels":["reserved:world"],"srcIdentity":2,"dstEpID":0,"dstEpLabels":["reserved:world"],"dstIdentity":2,"verdict":"Forwarded","http":{"Code":0,"Method":"GET","URL":{"Scheme":"http","Opaque":"","User":null,"Host":"10.50.50.169","Path":"/details/1","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"Protocol":"HTTP/1.1","Headers":{":scheme":["http"],"Accept":["*/*"],"User-Agent":["curl/7.87.0"],"X-Request-Id":["cf22128c-32cc-4c7b-bfc7-9962704c1bd4"]},"MissingHeaders":{},"RejectedHeaders":{}}}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}
{"cpu":"CPU 02:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.119:9080 dst=10.51.2.235:39775"}
{"cpu":"CPU 02:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=1"}
{"cpu":"CPU 02:","type":"debug","message":"CT entry found lifetime=37058, revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"CT verdict: Reply, revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 02:","type":"trace","mark":"0xa8cd380a","ifindex":"net0","state":"reply","observationPoint":"to-network","traceSummary":"-\u003e network","source":3734,"bytes":246,"srcLabel":46123,"dstLabel":8,"dstID":0,"summary":{"ethernet":"Ethernet\t{Contents=[..14..] Payload=[..118..] SrcMAC=92:be:64:23:e3:e5 DstMAC=52:98:da:91:c8:30 EthernetType=IPv4 Length=0}","ipv4":"IPv4\t{Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=232 Id=15526 Flags=DF FragOffset=0 TTL=63 Protocol=TCP Checksum=58530 SrcIP=10.51.2.119 DstIP=10.51.2.235 Options=[] Padding=[]}","tcp":"TCP\t{Contents=[..32..] Payload=[..66..] SrcPort=9080(glrpc) DstPort=39775 Seq=232423254 Ack=3072567909 DataOffset=8 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=508 Checksum=6818 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:2273117108/208856449 0x877d03b40c72e581)] Padding=[]}","l2":{"src":"92:be:64:23:e3:e5","dst":"52:98:da:91:c8:30"},"l3":{"src":"10.51.2.119","dst":"10.51.2.235"},"l4":{"src":"9080","dst":"39775"}}}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.51.2.119 to identity=46123"}
{"cpu":"CPU 02:","type":"debug","message":"Inheriting identity=2 from stack"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 02:","type":"debug","message":"Attempting local delivery for container id 3734 from seclabel 8"}
{"cpu":"CPU 02:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.235:39775 dst=10.51.2.119:9080"}
{"cpu":"CPU 02:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=0"}
{"cpu":"CPU 02:","type":"debug","message":"CT entry found lifetime=37058, revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"CT verdict: Established, revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.119:9080 dst=10.51.2.235:39775"}
{"cpu":"CPU 02:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=1"}
{"cpu":"CPU 02:","type":"debug","message":"CT entry found lifetime=37058, revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"CT verdict: Reply, revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.51.2.119 to identity=46123"}
{"cpu":"CPU 02:","type":"debug","message":"Inheriting identity=2 from stack"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.51.2.235 to identity=8"}
{"cpu":"CPU 02:","type":"debug","message":"Attempting local delivery for container id 3734 from seclabel 8"}
{"cpu":"CPU 02:","type":"debug","message":"Conntrack lookup 1/2: src=10.51.2.235:39775 dst=10.51.2.119:9080"}
{"cpu":"CPU 02:","type":"debug","message":"Conntrack lookup 2/2: nexthdr=6 flags=0"}
{"cpu":"CPU 02:","type":"debug","message":"CT entry found lifetime=37058, revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"CT verdict: Established, revnat=0"}
{"type":"logRecord","observationPoint":"Egress","flowType":"Response","l7Proto":"http","srcEpID":0,"srcEpLabels":["reserved:world"],"srcIdentity":2,"dstEpID":0,"dstEpLabels":["reserved:world"],"dstIdentity":2,"verdict":"Forwarded","http":{"Code":200,"Method":"GET","URL":{"Scheme":"http","Opaque":"","User":null,"Host":"10.50.50.169","Path":"/details/1","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"Protocol":"HTTP/1.1","Headers":{"Connection":["Keep-Alive"],"Content-Length":["178"],"Content-Type":["application/json"],"Date":["Sun, 12 Mar 2023 16:58:40 GMT"],"Server":["WEBrick/1.6.0 (Ruby/2.7.1/2020-03-31)"],"X-Envoy-Upstream-Service-Time":["2"],"X-Request-Id":["cf22128c-32cc-4c7b-bfc7-9962704c1bd4"]},"MissingHeaders":{},"RejectedHeaders":{}}}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=192.168.10.100 to identity=2"}
{"cpu":"CPU 02:","type":"debug","message":"CT entry found lifetime=49730, revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"CT created 1/2: sport=0 dport=49730 nexthdr=0 flags= revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=192.168.10.100 to identity=2"}
{"cpu":"CPU 01:","type":"debug","message":"CT entry found lifetime=49730, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT created 1/2: sport=0 dport=49730 nexthdr=0 flags= revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}
{"cpu":"CPU 01:","type":"debug","message":"Successfully mapped addr=192.168.10.100 to identity=2"}
{"cpu":"CPU 01:","type":"debug","message":"CT entry found lifetime=49730, revnat=0"}
{"cpu":"CPU 01:","type":"debug","message":"CT created 1/2: sport=0 dport=49730 nexthdr=0 flags= revnat=0"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}
{"cpu":"CPU 02:","type":"debug","message":"Successfully mapped addr=10.0.1.2 to identity=7"}


#### Hubble observe ####

{"flow":{"time":"2023-03-12T16:58:40.629048876Z","verdict":"TRACED","IP":{"source":"192.168.10.100","destination":"10.0.2.2","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":36698}},"source":{"identity":2,"labels":["reserved:world"]},"destination":{"identity":1,"labels":["reserved:host"]},"Type":"SOCK","node_name":"kind-bazzinga/bazzinga-worker","event_type":{"type":7,"sub_type":3},"sock_xlate_point":"SOCK_XLATE_POINT_PRE_DIRECTION_REV","socket_cookie":"77536","cgroup_id":"84198","Summary":"TCP"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.629048876Z"}
{"flow":{"time":"2023-03-12T16:58:40.629525918Z","verdict":"FORWARDED","ethernet":{"source":"52:98:da:91:c8:30","destination":"92:be:64:23:e3:e5"},"IP":{"source":"10.51.2.235","destination":"10.51.2.119","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":39775,"destination_port":9080,"flags":{"SYN":true}}},"source":{"identity":8,"labels":["reserved:ingress"]},"destination":{"ID":3734,"identity":46123,"namespace":"gwapi","labels":["k8s:app=details","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=gwapi","k8s:io.cilium.k8s.policy.cluster=kind-bazzinga","k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-details","k8s:io.kubernetes.pod.namespace=gwapi","k8s:version=v1"],"pod_name":"details-v1-7586964646-rk5w6","workloads":[{"name":"details-v1","kind":"Deployment"}]},"Type":"L3_L4","node_name":"kind-bazzinga/bazzinga-worker","event_type":{"type":4},"traffic_direction":"INGRESS","trace_observation_point":"TO_ENDPOINT","is_reply":false,"interface":{"index":9,"name":"lxce564053d40dc"},"Summary":"TCP Flags: SYN"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.629525918Z"}
{"flow":{"time":"2023-03-12T16:58:40.629528293Z","verdict":"FORWARDED","ethernet":{"source":"92:be:64:23:e3:e5","destination":"52:98:da:91:c8:30"},"IP":{"source":"10.51.2.119","destination":"10.51.2.235","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":9080,"destination_port":39775,"flags":{"SYN":true,"ACK":true}}},"source":{"ID":3734,"identity":46123,"namespace":"gwapi","labels":["k8s:app=details","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=gwapi","k8s:io.cilium.k8s.policy.cluster=kind-bazzinga","k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-details","k8s:io.kubernetes.pod.namespace=gwapi","k8s:version=v1"],"pod_name":"details-v1-7586964646-rk5w6","workloads":[{"name":"details-v1","kind":"Deployment"}]},"destination":{"identity":8,"labels":["reserved:ingress"]},"Type":"L3_L4","node_name":"kind-bazzinga/bazzinga-worker","reply":true,"event_type":{"type":4,"sub_type":11},"traffic_direction":"INGRESS","trace_observation_point":"TO_NETWORK","is_reply":true,"interface":{"index":337,"name":"net0"},"Summary":"TCP Flags: SYN, ACK"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.629528293Z"}
{"flow":{"time":"2023-03-12T16:58:40.629571751Z","verdict":"FORWARDED","ethernet":{"source":"52:98:da:91:c8:30","destination":"92:be:64:23:e3:e5"},"IP":{"source":"10.51.2.235","destination":"10.51.2.119","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":39775,"destination_port":9080,"flags":{"ACK":true}}},"source":{"identity":8,"labels":["reserved:ingress"]},"destination":{"ID":3734,"identity":46123,"namespace":"gwapi","labels":["k8s:app=details","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=gwapi","k8s:io.cilium.k8s.policy.cluster=kind-bazzinga","k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-details","k8s:io.kubernetes.pod.namespace=gwapi","k8s:version=v1"],"pod_name":"details-v1-7586964646-rk5w6","workloads":[{"name":"details-v1","kind":"Deployment"}]},"Type":"L3_L4","node_name":"kind-bazzinga/bazzinga-worker","event_type":{"type":4},"traffic_direction":"INGRESS","trace_observation_point":"TO_ENDPOINT","is_reply":false,"interface":{"index":9,"name":"lxce564053d40dc"},"Summary":"TCP Flags: ACK"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.629571751Z"}
{"flow":{"time":"2023-03-12T16:58:40.629809251Z","verdict":"TRACED","IP":{"source":"10.51.2.235","destination":"10.51.2.119","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":39775}},"source":{"identity":8,"labels":["reserved:ingress"]},"destination":{"ID":3734,"identity":46123,"namespace":"gwapi","labels":["k8s:app=details","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=gwapi","k8s:io.cilium.k8s.policy.cluster=kind-bazzinga","k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-details","k8s:io.kubernetes.pod.namespace=gwapi","k8s:version=v1"],"pod_name":"details-v1-7586964646-rk5w6","workloads":[{"name":"details-v1","kind":"Deployment"}]},"Type":"SOCK","node_name":"kind-bazzinga/bazzinga-worker","event_type":{"type":7,"sub_type":3},"sock_xlate_point":"SOCK_XLATE_POINT_PRE_DIRECTION_REV","socket_cookie":"73342","cgroup_id":"86200","Summary":"TCP"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.629809251Z"}
{"flow":{"time":"2023-03-12T16:58:40.629829626Z","verdict":"FORWARDED","ethernet":{"source":"52:98:da:91:c8:30","destination":"92:be:64:23:e3:e5"},"IP":{"source":"10.51.2.235","destination":"10.51.2.119","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":39775,"destination_port":9080,"flags":{"PSH":true,"ACK":true}}},"source":{"identity":8,"labels":["reserved:ingress"]},"destination":{"ID":3734,"identity":46123,"namespace":"gwapi","labels":["k8s:app=details","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=gwapi","k8s:io.cilium.k8s.policy.cluster=kind-bazzinga","k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-details","k8s:io.kubernetes.pod.namespace=gwapi","k8s:version=v1"],"pod_name":"details-v1-7586964646-rk5w6","workloads":[{"name":"details-v1","kind":"Deployment"}]},"Type":"L3_L4","node_name":"kind-bazzinga/bazzinga-worker","event_type":{"type":4},"traffic_direction":"INGRESS","trace_observation_point":"TO_ENDPOINT","is_reply":false,"interface":{"index":9,"name":"lxce564053d40dc"},"Summary":"TCP Flags: ACK, PSH"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.629829626Z"}
{"flow":{"time":"2023-03-12T16:58:40.630267751Z","verdict":"FORWARDED","IP":{"source":"192.168.10.100","destination":"10.50.50.169","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":36698,"destination_port":80}},"source":{"identity":2,"labels":["reserved:world"]},"destination":{"identity":2,"labels":["reserved:world"]},"Type":"L7","node_name":"kind-bazzinga/bazzinga-worker","l7":{"type":"REQUEST","http":{"method":"GET","url":"http://10.50.50.169/details/1","protocol":"HTTP/1.1","headers":[{"key":":scheme","value":"http"},{"key":"Accept","value":"*/*"},{"key":"User-Agent","value":"curl/7.87.0"},{"key":"X-Request-Id","value":"cf22128c-32cc-4c7b-bfc7-9962704c1bd4"}]}},"event_type":{"type":129},"destination_service":{"name":"cilium-gateway-my-gateway","namespace":"gwapi"},"traffic_direction":"EGRESS","is_reply":false,"Summary":"HTTP/1.1 GET http://10.50.50.169/details/1"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.630267751Z"}
{"flow":{"time":"2023-03-12T16:58:40.630878918Z","verdict":"TRACED","IP":{"source":"10.51.2.235","destination":"10.51.2.119","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":39775}},"source":{"identity":8,"labels":["reserved:ingress"]},"destination":{"ID":3734,"identity":46123,"namespace":"gwapi","labels":["k8s:app=details","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=gwapi","k8s:io.cilium.k8s.policy.cluster=kind-bazzinga","k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-details","k8s:io.kubernetes.pod.namespace=gwapi","k8s:version=v1"],"pod_name":"details-v1-7586964646-rk5w6","workloads":[{"name":"details-v1","kind":"Deployment"}]},"Type":"SOCK","node_name":"kind-bazzinga/bazzinga-worker","event_type":{"type":7,"sub_type":3},"sock_xlate_point":"SOCK_XLATE_POINT_PRE_DIRECTION_REV","socket_cookie":"73342","cgroup_id":"86200","Summary":"TCP"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.630878918Z"}
{"flow":{"time":"2023-03-12T16:58:40.631460043Z","verdict":"TRACED","IP":{"source":"10.51.2.235","destination":"10.51.2.119","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":39775}},"source":{"identity":8,"labels":["reserved:ingress"]},"destination":{"ID":3734,"identity":46123,"namespace":"gwapi","labels":["k8s:app=details","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=gwapi","k8s:io.cilium.k8s.policy.cluster=kind-bazzinga","k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-details","k8s:io.kubernetes.pod.namespace=gwapi","k8s:version=v1"],"pod_name":"details-v1-7586964646-rk5w6","workloads":[{"name":"details-v1","kind":"Deployment"}]},"Type":"SOCK","node_name":"kind-bazzinga/bazzinga-worker","event_type":{"type":7,"sub_type":3},"sock_xlate_point":"SOCK_XLATE_POINT_PRE_DIRECTION_REV","socket_cookie":"73342","cgroup_id":"86200","Summary":"TCP"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.631460043Z"}
{"flow":{"time":"2023-03-12T16:58:40.632109001Z","verdict":"FORWARDED","ethernet":{"source":"92:be:64:23:e3:e5","destination":"52:98:da:91:c8:30"},"IP":{"source":"10.51.2.119","destination":"10.51.2.235","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":9080,"destination_port":39775,"flags":{"PSH":true,"ACK":true}}},"source":{"ID":3734,"identity":46123,"namespace":"gwapi","labels":["k8s:app=details","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=gwapi","k8s:io.cilium.k8s.policy.cluster=kind-bazzinga","k8s:io.cilium.k8s.policy.serviceaccount=bookinfo-details","k8s:io.kubernetes.pod.namespace=gwapi","k8s:version=v1"],"pod_name":"details-v1-7586964646-rk5w6","workloads":[{"name":"details-v1","kind":"Deployment"}]},"destination":{"identity":8,"labels":["reserved:ingress"]},"Type":"L3_L4","node_name":"kind-bazzinga/bazzinga-worker","reply":true,"event_type":{"type":4,"sub_type":11},"traffic_direction":"INGRESS","trace_observation_point":"TO_NETWORK","is_reply":true,"interface":{"index":337,"name":"net0"},"Summary":"TCP Flags: ACK, PSH"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.632109001Z"}
{"flow":{"time":"2023-03-12T16:58:40.632480335Z","verdict":"FORWARDED","IP":{"source":"10.50.50.169","destination":"192.168.10.100","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":36698}},"source":{"identity":2,"labels":["reserved:world"]},"destination":{"identity":2,"labels":["reserved:world"]},"Type":"L7","node_name":"kind-bazzinga/bazzinga-worker","l7":{"type":"RESPONSE","latency_ns":"3195042","http":{"code":200,"method":"GET","url":"http://10.50.50.169/details/1","protocol":"HTTP/1.1","headers":[{"key":"Connection","value":"Keep-Alive"},{"key":"Content-Length","value":"178"},{"key":"Content-Type","value":"application/json"},{"key":"Date","value":"Sun, 12 Mar 2023 16:58:40 GMT"},{"key":"Server","value":"WEBrick/1.6.0 (Ruby/2.7.1/2020-03-31)"},{"key":"X-Envoy-Upstream-Service-Time","value":"2"},{"key":"X-Request-Id","value":"cf22128c-32cc-4c7b-bfc7-9962704c1bd4"}]}},"reply":true,"event_type":{"type":129},"source_service":{"name":"cilium-gateway-my-gateway","namespace":"gwapi"},"traffic_direction":"EGRESS","is_reply":true,"Summary":"HTTP/1.1 200 3ms (GET http://10.50.50.169/details/1)"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:40.632480335Z"}
{"flow":{"time":"2023-03-12T16:58:43.741502044Z","verdict":"FORWARDED","ethernet":{"source":"ea:a7:10:36:3e:b6","destination":"0a:d9:d6:b9:d6:a3"},"IP":{"source":"10.51.2.57","destination":"10.0.3.2","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":4240,"destination_port":52914,"flags":{"ACK":true}}},"source":{"ID":2475,"identity":4,"labels":["reserved:health"]},"destination":{"identity":6,"labels":["reserved:remote-node"]},"Type":"L3_L4","node_name":"kind-bazzinga/bazzinga-worker","reply":true,"event_type":{"type":4,"sub_type":11},"traffic_direction":"INGRESS","trace_observation_point":"TO_NETWORK","is_reply":true,"interface":{"index":337,"name":"net0"},"Summary":"TCP Flags: ACK"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:43.741502044Z"}
{"flow":{"time":"2023-03-12T16:58:43.741896586Z","verdict":"FORWARDED","ethernet":{"source":"0a:d9:d6:b9:d6:a3","destination":"ea:a7:10:36:3e:b6"},"IP":{"source":"10.0.3.2","destination":"10.51.2.57","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52914,"destination_port":4240,"flags":{"ACK":true}}},"source":{"identity":2,"labels":["reserved:world"]},"destination":{"ID":2475,"identity":4,"labels":["reserved:health"]},"Type":"L3_L4","node_name":"kind-bazzinga/bazzinga-worker","event_type":{"type":4},"traffic_direction":"INGRESS","trace_observation_point":"TO_ENDPOINT","is_reply":false,"interface":{"index":7,"name":"lxc_health"},"Summary":"TCP Flags: ACK"},"node_name":"kind-bazzinga/bazzinga-worker","time":"2023-03-12T16:58:43.741896586Z"}

Anything else?

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/agentCilium agent related.area/servicemeshGH issues or PRs regarding servicemeshfeature/k8s-gateway-apifeature/k8s-ingresskind/community-reportThis was reported by a user in the Cilium community, eg via Slack.kind/enhancementThis would improve or streamline existing functionality.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions