ig nodeLabels not passed to kubernetes nodes in Hetzner #16159
Description
/kind bug
1. What kops version are you running? The command kops version, will display
this information.
v1.28.1
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
v1.27.8
3. What cloud provider are you using?
Hetzner
4. What commands did you run? What is the simplest way to reproduce this issue?
Create the cluster
kops create cluster --name=my-cluster.lukasre.k8s.local \
--ssh-public-key=path-to-pub --cloud=hetzner --zones=fsn1 \
--image=ubuntu-20.04 --networking=calico --network-cidr=10.10.0.0/16Add a new instance group with different node labels
kops create ig nodes-immich-fsn1 --subnet fsn1Edit the instance group with the following config:
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: "2023-12-06T20:46:51Z"
generation: 3
labels:
kops.k8s.io/cluster: my-cluster.lukasre.k8s.local
name: nodes-immich-fsn1
spec:
image: ubuntu-22.04
kubelet:
anonymousAuth: false
nodeLabels:
lukasre.ch/instancetype: immich
node-role.kubernetes.io/node: ""
machineType: cx21
manager: CloudGroup
maxSize: 1
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: nodes-immich-fsn1
lukasre.ch/instancetype: immich
role: Node
subnets:
- fsn1Update the cluster (including forcing a rolling update) with
kops update cluster --yes
kops rolling-update cluster --yes --force5. What happened after the commands executed?
Commands are successful, but node labels are not added.
The yaml representation of the newly created node is the following (only metadata)
apiVersion: v1
kind: Node
metadata:
annotations:
alpha.kubernetes.io/provided-node-ip: 10.10.0.7
csi.volume.kubernetes.io/nodeid: '{"csi.hetzner.cloud":"40260390"}'
node.alpha.kubernetes.io/ttl: "0"
projectcalico.org/IPv4Address: 10.10.0.7/32
projectcalico.org/IPv4IPIPTunnelAddr: x.x.x.x
volumes.kubernetes.io/controller-managed-attach-detach: "true"
creationTimestamp: "2023-12-07T08:52:42Z"
labels:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/instance-type: cx21
beta.kubernetes.io/os: linux
csi.hetzner.cloud/location: fsn1
failure-domain.beta.kubernetes.io/region: fsn1
failure-domain.beta.kubernetes.io/zone: fsn1-dc14
kubernetes.io/arch: amd64
kubernetes.io/hostname: nodes-immich-fsn1-67573cda4d994baa
kubernetes.io/os: linux
node-role.kubernetes.io/node: ""
node.kubernetes.io/instance-type: cx21
topology.kubernetes.io/region: fsn1
topology.kubernetes.io/zone: fsn1-dc14
name: nodes-immich-fsn1-67573cda4d994baa
resourceVersion: "21804205"
uid: f730e308-5bb9-49f3-b530-91f7a74b698c6. What did you expect to happen?
The node labels specified in the instance group
kops.k8s.io/instancegroup: nodes-immich-fsn1
lukasre.ch/instancetype: immich
are not added to the nodes
7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.
piVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
creationTimestamp: "2023-09-19T18:44:55Z"
generation: 2
name: my-cluster.lukasre.k8s.local
spec:
api:
loadBalancer:
type: Public
authorization:
rbac: {}
channel: stable
cloudProvider: hetzner
configBase: <configBase>
etcdClusters:
- cpuRequest: 200m
etcdMembers:
- instanceGroup: control-plane-fsn1
name: etcd-1
manager:
backupRetentionDays: 90
memoryRequest: 100Mi
name: main
- cpuRequest: 100m
etcdMembers:
- instanceGroup: control-plane-fsn1
name: etcd-1
manager:
backupRetentionDays: 90
memoryRequest: 100Mi
name: events
iam:
allowContainerRegistry: true
legacy: false
kubelet:
anonymousAuth: false
kubernetesApiAccess:
- 0.0.0.0/0
- ::/0
kubernetesVersion: 1.27.8
networkCIDR: 10.10.0.0/16
networking:
calico: {}
nonMasqueradeCIDR: 100.64.0.0/10
sshAccess:
- 0.0.0.0/0
- ::/0
subnets:
- name: fsn1
type: Public
zone: fsn1
topology:
dns:
type: None
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: "2023-09-19T18:44:55Z"
labels:
kops.k8s.io/cluster: my-cluster.lukasre.k8s.local
name: control-plane-fsn1
spec:
image: ubuntu-20.04
machineType: cx21
maxSize: 1
minSize: 1
role: Master
subnets:
- fsn1
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: "2023-09-19T18:44:55Z"
generation: 2
labels:
kops.k8s.io/cluster: my-cluster.lukasre.k8s.local
name: nodes-fsn1
spec:
image: ubuntu-20.04
machineType: cx21
maxSize: 2
minSize: 2
role: Node
subnets:
- fsn1
---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: "2023-12-06T20:46:51Z"
generation: 3
labels:
kops.k8s.io/cluster: my-cluster.lukasre.k8s.local
name: nodes-immich-fsn1
spec:
image: ubuntu-22.04
kubelet:
anonymousAuth: false
nodeLabels:
lukasre.ch/instancetype: immich
node-role.kubernetes.io/node: ""
machineType: cx21
manager: CloudGroup
maxSize: 1
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: nodes-immich-fsn1
lukasre.ch/instancetype: immich
role: Node
subnets:
- fsn1
8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
9. Anything else do we need to know?
I looked at some existing issues and found #15090 and it seems it might be a similar issue:
If you compare how labels are generated for OpenStack here and for hetzner here it seems that the labels are not passed to the nodeIdentity.Info object.