Latest stable channel update references breaks 1.26 clusters #16956
Description
/kind bug
1. What kops version are you running? The command kops version, will display
this information.
1.27.3
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
1.26.15
3. What cloud provider are you using?
AWS
4. What commands did you run? What is the simplest way to reproduce this issue?
kops toolbox template --template template.yaml where template.yaml is the following:
kubernetesVersion: {{ ChannelRecommendedKubernetesUpgradeVersion "1.26" }}
5. What happened after the commands executed?
Returns an invalid version of Kubernetes:
kubernetesVersion: 1.26.16
(as defined in https://github.com/kubernetes/kops/blob/master/channels/stable#L119, which was updated 5 days ago with d7a8445)
6. What did you expect to happen?
Returns the latest version of 1.26, which is 1.26.15 as per https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md:
kubernetesVersion: 1.26.15
7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.
8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
Running a kops update cluster -v 10 with the new, invalid, Kubernetes version gives the following:
I1125 12:25:52.578392 80003 builder.go:338] Trying to read hash fie: "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256"
I1125 12:25:52.578589 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256
I1125 12:25:52.952929 80003 context.go:324] hit maximum retries 1 with error file does not exist
I1125 12:25:52.952976 80003 builder.go:342] Unable to read hash file "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256": file does not exist
I1125 12:25:52.952995 80003 builder.go:357] Unable to read new sha256 hash file (is this an older/unsupported kubernetes release?)
I1125 12:25:52.953016 80003 builder.go:338] Trying to read hash fie: "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1"
I1125 12:25:52.953286 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1
I1125 12:25:53.133946 80003 context.go:324] hit maximum retries 1 with error file does not exist
I1125 12:25:53.133995 80003 builder.go:342] Unable to read hash file "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1": file does not exist
I1125 12:25:53.134023 80003 builder.go:338] Trying to read hash fie: "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256"
I1125 12:25:53.134045 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256
I1125 12:25:53.325239 80003 context.go:320] retrying after error file does not exist
I1125 12:25:53.826151 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256
I1125 12:25:53.993253 80003 context.go:324] hit maximum retries 2 with error file does not exist
I1125 12:25:53.993287 80003 builder.go:342] Unable to read hash file "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256": file does not exist
I1125 12:25:53.993297 80003 builder.go:357] Unable to read new sha256 hash file (is this an older/unsupported kubernetes release?)
I1125 12:25:53.993309 80003 builder.go:338] Trying to read hash fie: "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1"
I1125 12:25:53.993321 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1
I1125 12:25:54.183914 80003 context.go:320] retrying after error file does not exist
I1125 12:25:54.684221 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1
I1125 12:25:54.887056 80003 context.go:324] hit maximum retries 2 with error file does not exist
I1125 12:25:54.887095 80003 builder.go:342] Unable to read hash file "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1": file does not exist
I1125 12:25:54.887130 80003 builder.go:338] Trying to read hash fie: "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256"
I1125 12:25:54.887148 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256
I1125 12:25:55.080773 80003 context.go:320] retrying after error file does not exist
I1125 12:25:55.581939 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256
I1125 12:25:55.757265 80003 context.go:320] retrying after error file does not exist
I1125 12:25:56.758470 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256
I1125 12:25:56.921471 80003 context.go:324] hit maximum retries 3 with error file does not exist
I1125 12:25:56.921507 80003 builder.go:342] Unable to read hash file "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha256": file does not exist
I1125 12:25:56.921517 80003 builder.go:357] Unable to read new sha256 hash file (is this an older/unsupported kubernetes release?)
I1125 12:25:56.921529 80003 builder.go:338] Trying to read hash fie: "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1"
I1125 12:25:56.921540 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1
I1125 12:25:57.076875 80003 context.go:320] retrying after error file does not exist
I1125 12:25:57.577740 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1
I1125 12:25:57.734164 80003 context.go:320] retrying after error file does not exist
I1125 12:25:58.735313 80003 context.go:252] Performing HTTP request: GET https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1
I1125 12:25:58.908778 80003 context.go:324] hit maximum retries 3 with error file does not exist
I1125 12:25:58.908808 80003 builder.go:342] Unable to read hash file "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet.sha1": file does not exist
Error: cannot determine hash for "https://dl.k8s.io/release/v1.26.16/bin/linux/amd64/kubelet" (have you specified a valid file location?)
9. Anything else do we need to know?
Seems like it's coming from #16947 and then was promoted with #16952