Skip to content

Container Cluster stuck in non-ready state because channel update rejected  #194

@jlewi

Description

@jlewi

Describe the bug
A clear and concise description of what the bug is.

ConfigConnector Version
Run the following command to get the current ConfigConnector version

kubectl get ns cnrm-system -o jsonpath='{.metadata.annotations.cnrm\.cloud\.google\.com/version}' 

cnrm.cloud.google.com/version: 1.9.1

To Reproduce
Steps to reproduce the behavior:

  1. Create a ContainerCluster CNRM resource setting the release channel for the cluster to get
    the status

  2. Apply the resource to create the cluster

  3. Reapply the resource

  4. Container cluster reports

      - lastTransitionTime: "2020-05-29T00:15:38Z"
       message: 'Update call failed: the desired mutation for the following field(s)
         is invalid: [releaseChannel.0.Channel]'
       reason: UpdateFailed
       status: "False"
       type: Ready
    

The release channel shouldn't be changing.

I suspect this an issue in the update logic since there are some restrictions about mutations to
release channel
https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels

YAML snippets:

apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerCluster
metadata:
  clusterName: gcp-private-dev/us-central1/jl-0601
  labels:
    kf-name: gcp-private-0527
    mesh_id: gcp-private-dev_us-central1_jl-0601
  name: jl-0601
  namespace: gcp-private-dev
spec:
  clusterAutoscaling:
    autoProvisioningDefaults:
      oauthScopes:
      - https://www.googleapis.com/auth/logging.write
      - https://www.googleapis.com/auth/monitoring
      - https://www.googleapis.com/auth/devstorage.read_only
      serviceAccountRef:
        name: jl-0601-vm
    enabled: true
    resourceLimits:
    - maximum: 128
      resourceType: cpu
    - maximum: 2000
      resourceType: memory
    - maximum: 16
      resourceType: nvidia-tesla-k80
  initialNodeCount: 2
  ipAllocationPolicy:
    clusterSecondaryRangeName: pods
    createSubnetwork: false
    servicesSecondaryRangeName: services
    useIpAliases: true
  location: us-central1
  loggingService: logging.googleapis.com/kubernetes
  monitoringService: monitoring.googleapis.com/kubernetes
  networkRef:
    name: jl-0601
  nodeConfig:
    machineType: n1-standard-8
    metadata:
      disable-legacy-endpoints: "true"
    oauthScopes:
    - https://www.googleapis.com/auth/logging.write
    - https://www.googleapis.com/auth/monitoring
    - https://www.googleapis.com/auth/devstorage.read_only
    serviceAccountRef:
      name: jl-0601-vm
    workloadMetadataConfig:
      nodeMetadata: GKE_METADATA_SERVER
  privateClusterConfig:
    enablePrivateEndpoint: false
    enablePrivateNodes: true
    masterIpv4CidrBlock: 172.16.0.32/28
  releaseChannel:
    channel: stable
  subnetworkRef:
    name: jl-0601
  workloadIdentityConfig:
    identityNamespace: gcp-private-dev.svc.id.goog

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions