External Workloads not working with non-default cluster_id #29355
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
Hi! I'm trying to set up External Workloads following the Install and configure Cilium on external workloads guide.
When I'm changing cluster.id, I can't connect VM to the service mesh.
Steps to reproduce:
cilium install --version 1.14.4 --set tunnel=vxlan --set cluster.id=11 --set cluster.name=cluster.local
cilium clustermesh enable --service-type NodePort --enable-external-workloads
cilium clustermesh vm create ts249 -n default --ipv4-alloc-cidr 10.192.1.0/30
cilium clustermesh vm install install-external-workload.shWith the default cluster.id, I am able to successfully connect VM to the cluster mesh.
Cilium Version
cilium-cli: v0.15.14 compiled with go1.21.4 on linux/amd64
cilium image (default): v1.14.2
cilium image (stable): v1.14.4
cilium image (running): 1.14.4
Kernel Version
Linux master 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Kubernetes Version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2
Sysdump
No response
Relevant log output
#Logs from the clustermesh-apiserver Pod
level=info msg="Upserting identity in etcd" identity=733573 subsys=clustermesh-apiserver
level=info msg="Upserting endpoint in etcd" endpoint=kube-system/dns-autoscaler-8576bb9f5b-qqszz ipAddr=10.0.2.146 subsys=clustermesh-apiserver
level=warning msg="Unable to unmarshal store value: {\"Name\":\"ts249\",\"Cluster\":\"\",\"IPAddresses\":null,\"IPv4AllocCIDR\":null,\"IPv4SecondaryAllocCIDRs\":null,\"IPv6AllocCIDR\":null,\"IPv6SecondaryAllocCIDRs\":null,\"IPv4HealthIP\":\"\",\"IPv6HealthIP\":\"\",\"IPv4IngressIP\":\"\",\"IPv6IngressIP\":\"\",\"ClusterID\":0,\"Source\":\"local\",\"EncryptionKey\":0,\"Labels\":null,\"Annotations\":null,\"NodeIdentity\":0,\"WireguardPubKey\":\"\"}" error="ClusterID 0 is reserved" eventType=create key=cilium/state/noderegister/v1/ts249 storeName=store-cilium/state/noderegister/v1 subsys=shared-store
level=warning msg="Unable to unmarshal store value: {\"Name\":\"ts249\",\"Cluster\":\"\",\"IPAddresses\":null,\"IPv4AllocCIDR\":null,\"IPv4SecondaryAllocCIDRs\":null,\"IPv6AllocCIDR\":null,\"IPv6SecondaryAllocCIDRs\":null,\"IPv4HealthIP\":\"\",\"IPv6HealthIP\":\"\",\"IPv4IngressIP\":\"\",\"IPv6IngressIP\":\"\",\"ClusterID\":0,\"Source\":\"local\",\"EncryptionKey\":0,\"Labels\":null,\"Annotations\":null,\"NodeIdentity\":0,\"WireguardPubKey\":\"\"}" error="ClusterID 0 is reserved" eventType=modify key=cilium/state/noderegister/v1/ts249 storeName=store-cilium/state/noderegister/v1 subsys=shared-store
#Logs from the VM
./install-external-workload.sh
Launching Cilium agent quay.io/cilium/cilium:v1.14.4@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e...
Unable to find image 'quay.io/cilium/cilium:v1.14.4@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e' locally
quay.io/cilium/cilium@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e: Pulling from cilium/cilium
2da7f86e18dc: Pull complete
2f85f16b0d22: Pull complete
a43301f52eec: Pull complete
2f0a070b113d: Pull complete
8e2c26839c4b: Pull complete
f5ffe5c32731: Pull complete
4be18d6a3596: Pull complete
Digest: sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e
Status: Downloaded newer image for quay.io/cilium/cilium@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e
d4b0bb8252491c5e9ef7b80a1fd6d57e6a61042c692c0f00445ebc3db974897b
Successfully copied 69.2MB to /usr/bin/cilium-dbg
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Cilium status:
Get "http://localhost/v1/healthz": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory
Is the agent running?
Restarting Cilium...
Shutting down running Cilium agent
cilium
Launching Cilium agent quay.io/cilium/cilium:v1.14.4@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e...
661a6baf8e02da8547664873bf151a52bea1f5883cdb51bef5bc75fb32574df9
Successfully copied 69.2MB to /usr/bin/cilium-dbg
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Waiting for Cilium daemon to come up...
Cilium status:
Get "http://localhost/v1/healthz": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory
Is the agent running?
Restarting Cilium...
Shutting down running Cilium agentAnything else?
Logs from successful run(with cluster_id: 0) clustermesh-apiserver Pod:
level=info msg="Upserting node in etcd" node=ts249 subsys=clustermesh-apiserver
level=info msg="Successfully created CiliumNode resource: {{ } {ts249 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[io.cilium.k8s.policy.cluster:cluster.local io.kubernetes.pod.name:ts249 io.kubernetes.pod.namespace:default] map[] [{cilium.io/v2 CiliumExternalWorkload ts249 cec3ba50-7e14-47f6-b59e-d25fbe0d027d <nil> <nil>}] [] []} { [{InternalIP 10.211.80.29} {InternalIP fc00::10ca:1} {CiliumInternalIP 10.192.1.2} {CiliumInternalIP f00d::a1d:0:0:10ad}] { } { } {0} { 0 0 0 <nil> [] map[] [] map[] map[] <nil> <nil> <nil>} {} { [] map[] [] map[]} {map[] {[] []} [10.192.1.0/30 f00d::a1d:0:0:0/96] 0 0 0 0 0 0} 1} {{map[]} {[]} {map[] map[] {} map[]} {map[]}}}" subsys=clustermesh-apiserver
level=info msg="Getting Node during an CEP update" subsys=clustermesh-apiserver
level=info msg="Getting CEP during an initialization" subsys=clustermesh-apiserver
level=info msg="Upserting node in etcd" node=ts249 subsys=clustermesh-apiserver
level=info msg="Successfully created CiliumEndpoint resource default/ts249: {\"metadata\":{\"name\":\"ts249\",\"namespace\":\"default\",\"creationTimestamp\":null,\"labels\":{\"name\":\"ts249\"},\"ownerReferences\":[{\"apiVersion\":\"cilium.io/v2\",\"kind\":\"CiliumNode\",\"name\":\"ts249\",\"uid\":\"d3e4c85d-2e3e-4165-8ae7-a347c0741d3f\"}]},\"status\":{\"encryption\":{}}}" subsys=clustermesh-apiserver
level=info msg="Returned CiliumEndpoint resource default/ts249: {\"metadata\":{\"name\":\"ts249\",\"namespace\":\"default\",\"uid\":\"21b40d2a-a0d0-4a6a-93f7-34d8b43204d6\",\"resourceVersion\":\"322675\",\"generation\":1,\"creationTimestamp\":\"2023-11-24T06:29:16Z\",\"labels\":{\"name\":\"ts249\"},\"ownerReferences\":[{\"apiVersion\":\"cilium.io/v2\",\"kind\":\"CiliumNode\",\"name\":\"ts249\",\"uid\":\"d3e4c85d-2e3e-4165-8ae7-a347c0741d3f\"}],\"managedFields\":[{\"manager\":\"clustermesh-apiserver\",\"operation\":\"Update\",\"apiVersion\":\"cilium.io/v2\",\"time\":\"2023-11-24T06:29:16Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:labels\":{\".\":{},\"f:name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"d3e4c85d-2e3e-4165-8ae7-a347c0741d3f\\\"}\":{}}},\"f:status\":{\".\":{},\"f:encryption\":{}}}}]},\"status\":{\"encryption\":{}}}" subsys=clustermesh-apiserver
level=info msg="Got Endpoint Identity: {14130 [k8s:io.cilium.k8s.policy.cluster=cluster.local k8s:io.kubernetes.pod.name=ts249 k8s:io.kubernetes.pod.namespace=default]}" subsys=clustermesh-apiserver
level=info msg="Successfully patched CiliumEndpoint resource: {{ } {ts249 default 21b40d2a-a0d0-4a6a-93f7-34d8b43204d6 322676 2 2023-11-24 06:29:16 +0000 UTC <nil> <nil> map[name:ts249] map[] [{cilium.io/v2 CiliumNode ts249 d3e4c85d-2e3e-4165-8ae7-a347c0741d3f <nil> <nil>}] [] [{clustermesh-apiserver Update cilium.io/v2 2023-11-24 06:29:16 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"d3e4c85d-2e3e-4165-8ae7-a347c0741d3f\\\"}\":{}}},\"f:status\":{\".\":{},\"f:encryption\":{},\"f:id\":{},\"f:identity\":{\".\":{},\"f:id\":{},\"f:labels\":{}},\"f:networking\":{\".\":{},\"f:addressing\":{},\"f:node\":{}},\"f:state\":{}}} }]} {1 [] <nil> <nil> 0xc0008db400 [] 0xc000976570 {0} <nil> <nil> ready []}}" subsys=clustermesh-apiserver
level=info msg="Upserting endpoint in etcd" endpoint=default/ts249 ipAddr=10.211.80.29 subsys=clustermesh-apiserver
level=info msg="Upserting endpoint in etcd" endpoint=default/ts249 ipAddr="fc00::10ca:1" subsys=clustermesh-apiserver
level=info msg="Upserting endpoint in etcd" endpoint=default/ts249 ipAddr=10.192.1.2 subsys=clustermesh-apiserver
level=info msg="Upserting endpoint in etcd" endpoint=default/ts249 ipAddr="f00d::a1d:0:0:10ad" subsys=clustermesh-apiserver
VM:
./install-external-workload.sh
Launching Cilium agent quay.io/cilium/cilium:v1.14.4@sha256:4981767b787c69126e190e33aee93d5a076639083c21f0e7c29596a519c64a2e...
a6b1c21056497991ea63148264a44d4a4b21b934054dee74faf546607578ea01
Successfully copied 69.2MB to /usr/bin/cilium-dbg
Waiting for Cilium daemon to come up...
Cilium status:
KVStore: Ok etcd: 1/1 connected, leases=0, lock lease-ID=7c028bffffd9cf71, has-quorum=true: https://clustermesh-apiserver.cilium.io:32379 - 3.5.4 (Leader)
Kubernetes: Disabled
Host firewall: Disabled
CNI Chaining: none
Cilium: Ok 1.14.4 (v1.14.4-87dd2b64)
NodeMonitor: Disabled
IPAM: IPv4: 1/2 allocated from 10.192.1.0/30, IPv6: 1/4294967294 allocated from f00d::a1d:0:0:0/96
IPv4 BIG TCP: Disabled
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: IPTables [IPv4: Enabled, IPv6: Enabled]
Controller Status: 15/15 healthy
Proxy Status: OK, ip 10.192.1.2, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range: min 256, max 65535
Hubble: Disabled
Encryption: Disabled
Cluster health: Probe disabled
Cilium successfully started!
Adding kube-dns IP 10.96.0.3 to /etc/resolv.conf
nameserver 10.96.0.3
nameserver 172.17.0.1
nameserver 127.0.0.1
search test
search default.svc.cluster.local svc.cluster.local cluster.local
Code of Conduct
- I agree to follow this project's Code of Conduct