Rancher Server Setup

• Rancher version: v2.7.0-rc6
• Installation option (Docker install/Helm Chart): Docker
• Proxy/Cert Details: self-signed

Information about the Cluster

• Kubernetes version: v1.21.14-eks-6d3986b
• Cluster Type (Local/Downstream): Downstream EKS cluster

User Information

• What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
  Standard User/Cluster Owner

Describe the bug
Unable to upgrade EKS clusters, w/ k8s 1.21 or below, on rancher 2.7

To Reproduce

1. Fresh install of rancher v2.6.9
2. As admin, create a standard user
3. As standard user, provision a downstream EKS cluster w/ k8s 1.21
4. Once Active, upgrade rancher to v2.7.0-rc6
5. Once upgrade is successful, Edit Config of EKS cluster and upgrade to k8s 1.23
6. Reproduced

Result
Unable to upgrade rancher:

• Because Amazon EKS runs a highly available control plane, you can update only one minor version at a time.
• 1.23 disabled w/ following message: minor version > 1 not allowed by EKS

Expected Result
Cluster should be able to upgrade to 1.23

Screenshots

Additional Context
Upon further testing, the following workaround was successful on 2.7.0-rc6 w/ EKS k8s 1.21:

1. Send API PUT request to <host_url>/v3/clusters/<cluster_id>
2. Request Body:

{
    "name": <REDACTED>,
    "eksConfig": {
        "kubernetesVersion": "1.22",
        "nodeGroups": [
            {
                "desiredSize": 2,
                "diskSize": 20,
                "ec2SshKey": "",
                "gpu": false,
                "imageId": "",
                "instanceType": "t3.xlarge",
                "labels": {},
                "launchTemplate": null,
                "maxSize": 2,
                "minSize": 2,
                "nodegroupName": <REDACTED>,
                "requestSpotInstances": false,
                "resourceTags": {},
                "spotInstanceTypes": [],
                "subnets": [
                    <REDACTED>,
                    <REDACTED>,
                    <REDACTED>
                ],
                "tags": {},
                "type": "/v3/schemas/nodeGroup",
                "userData": "",
                "version": "1.22"
            }
        ]
    }
} 

3. Result:

• Status code 200
• Cluster + nodes successfully upgrade to 1.22

4. Once upgraded to k8s 1.22, Edit Config of cluster and upgrade to 1.23 via Rancher UI
5. Cluster successfully upgrades to 1.23

Additional Workaround - [to be release-noted]

1. From AWS Console, upgrade EKS cluster from 1.21 => 1.22
2. Once upgraded + sync'd w/ Rancher, From Rancher UI, Edit Config and upgrade nodes to 1.22
3. Once nodes are upgraded to 1.22 and Active, From Rancher UI, Edit Config and upgrade cluster to 1.23
4. Once cluster is upgraded + Active, From Rancher UI, Edit Config and upgrade nodes to 1.23
5. Verified - EKS cluster + nodes successfully upgraded to 1.23 on rancher v2.7.0-rc6