Checklist:

• [y] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [y] I've included steps to reproduce the bug.
• [y] I've pasted the output of argocd version.

Describe the bug

I am seeing an error in Argo CD when upgrading a Helm chart from one version to another. The only difference between the Helm chart versions is that the new version removes the resources block from spec.template.spec.containers[0] in the Deployment object. I have noticed that removing other blocks (e.g. env) results in the same issue (so it is not just a resources problem).

The specific error is:

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error removing non config mutations for resource Deployment/test-app: error reverting webhook removed fields in predicted live resource: .spec.template.spec.containers: element 0: associative list with keys has an element that omits key field "name" (and doesn't have default value)

Additional details:

• The sync options we have enabled are just ServerSideApply=true.
• For compare options, ServerSideDiff is enforced on the server side by setting controller.diff.server.side: true in the argo-helm chart values.
• After a certain period of time (indeterminate - a few minutes?), the app status goes to Healthy/Synced/Sync OK, which initially suggests it "fixed itself", however, you then notice that the "Last Sync" version is still the old Helm chart version. Then, clicking Refresh will re-trigger the error to appear. Triggering a manual Sync appears to actually fix the issue.

To Reproduce

I have built a Helm chart to reproduce this, with two versions (0.0.1 and 0.0.2). Updating will not work, as you will see.

• Prerequisites:

  • Argo CD installed to the kube-system namespace.
  • Server-side diff enabled (controller.diff.server.side: true).
  • Another namespace created called bug (to host the k8s objects).

• Create a file called application.yaml to represent our Argo CD Application object:

  apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
    name: test-app
  spec:
    project: default
    source:
      repoURL: https://thecosmicfrog.github.io/helm-charts/
      path: .
      targetRevision: 0.0.1  # Baseline chart version:
                             # https://github.com/thecosmicfrog/helm-charts/tree/0.0.1/charts/test-app
      helm:
        releaseName: test-app
      chart: test-app
    destination:
      server: https://kubernetes.default.svc
      namespace: bug
    syncPolicy:
      automated:
        prune: true
        selfHeal: true
      syncOptions:
        - ServerSideApply=true
    revisionHistoryLimit: 10

• Apply this to the cluster:
  kubectl apply -n kube-system -f application.yaml

• Pods should come up without issue and everything should be correctly synced - as expected for a simple Deployment.

• Update application.yaml to bump the Helm chart to the new version:

  apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
    name: test-app
  spec:
    project: default
    source:
      repoURL: https://thecosmicfrog.github.io/helm-charts/
      path: .
  -   targetRevision: 0.0.1  # Baseline chart version:
  -                          # https://github.com/thecosmicfrog/helm-charts/tree/0.0.1/charts/test-app
  +   targetRevision: 0.0.2  # New chart with `spec.template.spec.containers[0].resources` block removed:
  +                          # https://github.com/thecosmicfrog/helm-charts/compare/0.0.1...0.0.2
      helm:
        releaseName: test-app
      chart: test-app
    destination:
      server: https://kubernetes.default.svc
      namespace: bug
    syncPolicy:
      automated:
        prune: true
        selfHeal: true
      syncOptions:
        - ServerSideApply=true
    revisionHistoryLimit: 10

• Apply this to the cluster:
  kubectl apply -n kube-system -f application.yaml

Expected behavior

The new chart version should install without error, as it is such a straightforward change (resources block removed from containers[0] block).

Actual behavior

Sync Status enters an Unknown state with the new chart version, and App Conditions displays 1 Error. That error is:

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error removing non config mutations for resource Deployment/test-app: error reverting webhook removed fields in predicted live resource: .spec.template.spec.containers: element 0: associative list with keys has an element that omits key field "name" (and doesn't have default value)

The only way to seemingly complete the update is to manually sync, which works without issue. We have Auto Sync enabled, so I'm not sure why that does not resolve the issue.

Screenshots

• Argo CD UI after applying chart 0.0.2:

  

• Several (5-ish?) minutes later - with no intervention from me - error "appears" to be resolved... but note the versions are not matching between both sync fields:

  

• Then, clicking Refresh...

  

• ...results in the same 1 Error outcome as before...

  

• ...and the Pods present on the cluster are still from the "old" (0.0.1) chart version:

  

• The only way to fix is to manually Sync:

  

• Which finally brings the app into sync at 0.0.2:

  

Version

2024/11/13 14:23:55 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
argocd: v2.13.0+347f221
  BuildDate: 2024-11-04T15:31:13Z
  GitCommit: 347f221adba5599ef4d5f12ee572b2c17d01db4d
  GitTreeState: clean
  GoVersion: go1.23.2
  Compiler: gc
  Platform: darwin/arm64
WARN[0001] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
argocd-server: v2.13.0+347f221
  BuildDate: 2024-11-04T12:09:06Z
  GitCommit: 347f221adba5599ef4d5f12ee572b2c17d01db4d
  GitTreeState: clean
  GoVersion: go1.23.1
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.4.3 2024-07-19T16:40:33Z
  Helm Version: v3.15.4+gfa9efb0
  Kubectl Version: v0.31.0
  Jsonnet Version: v0.20.0

Logs

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error removing non config mutations for resource Deployment/test-app: error reverting webhook removed fields in predicted live resource: .spec.template.spec.containers: element 0: associative list with keys has an element that omits key field "name" (and doesn't have default value)

Let me know if the above is enough information to reproduce the issue.

Thanks for your time - Aaron