Skip to content

hubble: fix endpoint cluster name #35415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Oct 30, 2024
Merged

hubble: fix endpoint cluster name #35415

merged 4 commits into from
Oct 30, 2024

Conversation

kaworu
Copy link
Member

@kaworu kaworu commented Oct 16, 2024

Add endpoint cluster name info for local endpoints, debug events, and L7 flows. Follow-up PR of #32313 which added cluster name to remote endpoints for L3L4 flows.

Locally tested with the Star Wars demo:

local endpoint 
% kubectl get pods --all-namespaces -o "custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName" | awk 'NR == 1 || /tiefighter|deathstar|cilium-[0-9a-z]+\s/ {print}'
NAMESPACE            NAME                                         NODE
default              deathstar-bf77cddc9-jpnls                    kind-worker
default              deathstar-bf77cddc9-q2q4p                    kind-control-plane
default              tiefighter                                    kind-worker
kube-system          cilium-5nsbl                                 kind-control-plane
kube-system          cilium-jxqxw                                 kind-worker
% kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
% kubectl exec -n kube-system cilium-jxqxw -c cilium-agent -- hubble observe flows --from-pod tiefighter --to-pod deathstar --type trace --first 1 -o json | jq
{
"flow": {
  "time": "2024-10-16T15:22:09.060066037Z",
  "uuid": "5e524183-b28d-4a01-a5dc-82128325e946",
  "verdict": "FORWARDED",
  "ethernet": {
    "source": "8a:37:ab:1d:35:f0",
    "destination": "ba:59:62:b6:08:01"
  },
  "IP": {
    "source": "10.244.1.249",
    "destination": "10.244.0.76",
    "ipVersion": "IPv4"
  },
  "l4": {
    "TCP": {
      "source_port": 56468,
      "destination_port": 80,
      "flags": {
        "SYN": true
      }
    }
  },
  "source": {
    "ID": 1250,
    "identity": 47651,
    "cluster_name": "kind-kind",
    "namespace": "default",
    "labels": [
      "k8s:app.kubernetes.io/name=tiefighter",
      "k8s:class=tiefighter",
      "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
      "k8s:io.cilium.k8s.policy.cluster=kind-kind",
      "k8s:io.cilium.k8s.policy.serviceaccount=default",
      "k8s:io.kubernetes.pod.namespace=default",
      "k8s:org=empire"
    ],
    "pod_name": "tiefighter"
  },
  "destination": {
    "identity": 21223,
    "cluster_name": "kind-kind",
    "namespace": "default",
    "labels": [
      "k8s:app.kubernetes.io/name=deathstar",
      "k8s:class=deathstar",
      "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
      "k8s:io.cilium.k8s.policy.cluster=kind-kind",
      "k8s:io.cilium.k8s.policy.serviceaccount=default",
      "k8s:io.kubernetes.pod.namespace=default",
      "k8s:org=empire"
    ],
    "pod_name": "deathstar-bf77cddc9-q2q4p"
  },
  "Type": "L3_L4",
  "node_name": "kind-kind/kind-worker",
  "node_labels": [
    "beta.kubernetes.io/arch=amd64",
    "beta.kubernetes.io/os=linux",
    "kubernetes.io/arch=amd64",
    "kubernetes.io/hostname=kind-worker",
    "kubernetes.io/os=linux"
  ],
  "event_type": {
    "type": 4,
    "sub_type": 4
  },
  "traffic_direction": "EGRESS",
  "trace_observation_point": "TO_OVERLAY",
  "trace_reason": "NEW",
  "is_reply": false,
  "interface": {
    "index": 4,
    "name": "cilium_vxlan"
  },
  "Summary": "TCP Flags: SYN"
},
"node_name": "kind-kind/kind-worker",
"time": "2024-10-16T15:22:09.060066037Z"
}
debug event 
% kubectl exec -it -n kube-system cilium-jxqxw -c cilium-agent -- cilium endpoint get -l k8s:app.kubernetes.io/name=tiefighter -o jsonpath='{[0].id}'
1250
% kubectl exec -it -n kube-system cilium-jxqxw -c cilium-agent -- cilium endpoint config 1250 Debug=true
Endpoint 1250 configuration updated successfully
% kubectl exec -n kube-system cilium-jxqxw -c cilium-agent -- hubble observe debug-events -f -o json | head -n1 | jq &
% kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
{
"debug_event": {
  "type": "DBG_CT_LOOKUP4_1",
  "source": {
    "ID": 1250,
    "identity": 47651,
    "cluster_name": "kind-kind",
    "namespace": "default",
    "labels": [
      "k8s:app.kubernetes.io/name=tiefighter",
      "k8s:class=tiefighter",
      "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
      "k8s:io.cilium.k8s.policy.cluster=kind-kind",
      "k8s:io.cilium.k8s.policy.serviceaccount=default",
      "k8s:io.kubernetes.pod.namespace=default",
      "k8s:org=empire"
    ],
    "pod_name": "tiefighter"
  },
  "hash": 635510294,
  "arg1": 4177654794,
  "arg2": 167796746,
  "arg3": 3522142,
  "message": "Conntrack lookup 1/2: src=10.244.1.249:48734 dst=10.96.0.10:53",
  "cpu": 0
},
"node_name": "kind-kind/kind-worker",
"time": "2024-10-16T15:47:21.803847602Z"
}
Ship landed
L7 
% kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
% kubectl exec -n kube-system cilium-jxqxw -c cilium-agent -- hubble observe flows --from-pod tiefighter --to-pod deathstar --type l7 --first 1 -o json | jq
{
"flow": {
  "time": "2024-10-16T15:50:01.789485280Z",
  "uuid": "52505121-81b9-4946-a02a-979c132b0096",
  "verdict": "FORWARDED",
  "IP": {
    "source": "10.244.1.249",
    "destination": "10.244.1.112",
    "ipVersion": "IPv4"
  },
  "l4": {
    "TCP": {
      "source_port": 36742,
      "destination_port": 80
    }
  },
  "source": {
    "ID": 1250,
    "identity": 47651,
    "cluster_name": "kind-kind",
    "namespace": "default",
    "labels": [
      "k8s:app.kubernetes.io/name=tiefighter",
      "k8s:class=tiefighter",
      "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
      "k8s:io.cilium.k8s.policy.cluster=kind-kind",
      "k8s:io.cilium.k8s.policy.serviceaccount=default",
      "k8s:io.kubernetes.pod.namespace=default",
      "k8s:org=empire"
    ],
    "pod_name": "tiefighter"
  },
  "destination": {
    "ID": 1947,
    "identity": 21223,
    "cluster_name": "kind-kind",
    "namespace": "default",
    "labels": [
      "k8s:app.kubernetes.io/name=deathstar",
      "k8s:class=deathstar",
      "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
      "k8s:io.cilium.k8s.policy.cluster=kind-kind",
      "k8s:io.cilium.k8s.policy.serviceaccount=default",
      "k8s:io.kubernetes.pod.namespace=default",
      "k8s:org=empire"
    ],
    "pod_name": "deathstar-bf77cddc9-jpnls",
    "workloads": [
      {
        "name": "deathstar",
        "kind": "Deployment"
      }
    ]
  },
  "Type": "L7",
  "node_name": "kind-kind/kind-worker",
  "node_labels": [
    "beta.kubernetes.io/arch=amd64",
    "beta.kubernetes.io/os=linux",
    "kubernetes.io/arch=amd64",
    "kubernetes.io/hostname=kind-worker",
    "kubernetes.io/os=linux"
  ],
  "l7": {
    "type": "REQUEST",
    "http": {
      "method": "POST",
      "url": "http://deathstar.default.svc.cluster.local/v1/request-landing",
      "protocol": "HTTP/1.1",
      "headers": [
        {
          "key": ":scheme",
          "value": "http"
        },
        {
          "key": "Accept",
          "value": "*/*"
        },
        {
          "key": "User-Agent",
          "value": "curl/7.88.1"
        },
        {
          "key": "X-Envoy-Internal",
          "value": "true"
        },
        {
          "key": "X-Request-Id",
          "value": "415b6302-ba7a-431e-93ab-59a9fd7d955a"
        }
      ]
    }
  },
  "event_type": {
    "type": 129
  },
  "traffic_direction": "INGRESS",
  "is_reply": false,
  "Summary": "HTTP/1.1 POST http://deathstar.default.svc.cluster.local/v1/request-landing"
},
"node_name": "kind-kind/kind-worker",
"time": "2024-10-16T15:50:01.789485280Z"
}

kaworu added 4 commits October 16, 2024 16:24
@kaworu
endpoint: change GetLabels() to return labels.Labels
0a7badc 
Preparation work to extract the endpoint cluster name from the returned
labels. From the caller's perspective, it is easier to find a well-known
label after this patch as we can index the returned labels.Labels map
instead of iterating over a string slice.

Signed-off-by: Alexandre Perrin <alex@isovalent.com>
@kaworu
hubble: add cluster name for local endpoints
bbbf9ab 
Before this patch, local endpoints would not have their cluster name set
in Hubble flows. Missed by 163c874.

Signed-off-by: Alexandre Perrin <alex@isovalent.com>
@kaworu
hubble: add cluster name to debug events endpoints
25b9b30 
Before this patch, Hubble debug events would not have their endpoint
cluster name set in Hubble flows.

Signed-off-by: Alexandre Perrin <alex@isovalent.com>
@kaworu
hubble: add cluster name to l7 flow endpoints
100655d 
Before this patch, endpoints would not have their cluster name set in
Hubble L7 flows. Missed by 163c874.

Signed-off-by: Alexandre Perrin <alex@isovalent.com>
@kaworu kaworu added kind/bug This is a bug in the Cilium logic. release-note/bug This PR fixes an issue in a previous release of Cilium. sig/hubble needs-backport/1.16 This PR / issue needs backporting to the v1.16 branch labels Oct 16, 2024
@kaworu kaworu self-assigned this Oct 16, 2024
@kaworu kaworu requested review from a team as code owners October 16, 2024 15:52
@kaworu kaworu requested review from rolinh and tklauser October 16, 2024 15:52
@kaworu kaworu mentioned this pull request Oct 16, 2024
Merged
@kaworu
Copy link
Member Author

kaworu commented Oct 16, 2024

/test

@kaworu
Copy link
Member Author

kaworu commented Oct 17, 2024

/ci-multi-pool

rolinh
rolinh approved these changes Oct 29, 2024
View reviewed changes
Copy link
Member

@rolinh rolinh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the fixes Alex 🙈

@rolinh rolinh enabled auto-merge October 30, 2024 08:45
@rolinh
Copy link
Member

rolinh commented Oct 30, 2024

/ci-gateway-api

@rolinh rolinh added this pull request to the merge queue Oct 30, 2024
@maintainer-s-little-helper maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Oct 30, 2024
Merged via the queue into cilium:main with commit 958537c Oct 30, 2024
74 checks passed
@kaworu kaworu deleted the pr/kaworu/hubble/fix-endpoint-cluster-name branch October 30, 2024 15:14
@joamaki joamaki mentioned this pull request Nov 5, 2024
Merged
23 tasks
@joamaki joamaki added backport-pending/1.16 The backport for Cilium 1.16.x for this PR is in progress. and removed needs-backport/1.16 This PR / issue needs backporting to the v1.16 branch labels Nov 5, 2024
@github-actions github-actions bot added backport-done/1.16 The backport for Cilium 1.16.x for this PR is done. and removed backport-pending/1.16 The backport for Cilium 1.16.x for this PR is in progress. labels Nov 7, 2024
@cilium-release-bot cilium-release-bot bot mentioned this pull request Nov 13, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tklauser tklauser tklauser approved these changes

@rolinh rolinh rolinh approved these changes

Assignees

@kaworu kaworu

Labels
backport-done/1.16 The backport for Cilium 1.16.x for this PR is done. kind/bug This is a bug in the Cilium logic. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/bug This PR fixes an issue in a previous release of Cilium.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kaworu @rolinh @tklauser @joamaki