BUG REPORT

tl;dr: any Chart containing a StatefulSet (any version) using a chart version in a volumeClaimTemplate can't be upgraded

This is the third and hopefully the last bug regarding "I can't upgrade my Release".

Note:

• for the selectors not being updated in apps/v1beta1 / extensions/v1beta1 problem, see Putting chart version in metadata.labels in a Deployment v1beta1 without selector prevents upgrade #7726.
• for the immutability of selectors apps/v1beta2 problem, see Putting chart version in spec.selector.matchLabels in apps/v1beta2 and later prevents ANY future upgrade of existing Release #7680.

Forewords

• Kubernetes API defines that some fields are immutable. In particular:
  • For a StatefulSet, everything except spec.replicas, spec.template, and spec.updateStrategy (i.e spec.selectorand spec.VolumeClaimTemplate is immutable)

The problem

In some charts, we set the chart version (directly or through the chart template) in spec.VolumeClaimTemplate.metadata.labels.

This causes helm to attempt to change immutable fields and any upgrade (even if it ONLY change the version in Chart.yaml) will fail.

IN OTHER WORDS, SOME CHARTS OF THIS REPOSITORY ARE DEEPLY BROKEN AND CAN'T BE UPGRADED, only installed once.

Impacted charts

At least, impacted charts are:

• stable/mariadb (fixed in [stable/mariadb] Fix issue with pvc on upgrades #7966)
• stable/redis (fixed in [stable/redis] Major version bump: Fix chart not being upgradable #7686)
• stable/rabbitmq (fixed in [stable/rabbitmq] Fix chart not being upgradable by removing mutable label from VolumeClaimTemplate #7687)
• incubator/couchdb (PR in Allow statefulSet updates #8527)
• incubator/patroni (PR in [incubator/patroni] Fix chart not being upgradable because of spec.VolumeClaimTemplate.metadata.labels being immutable #8594)

How to reproduce (using stable/mariadb as an example)

Let's install the stable/mariadb chart using git, change the version in chart.yaml (example: 1.2.3 -> 1.2.4) without changing anything else, and try to upgrade. See it fail:

cd stable/mariadb
helm install --name test .
vim Chart.yaml
helm upgrade test .

Because of spec.volumeClaimTemplate.0.metadata.labels in templates/master-statefulset.yaml containing:

chart: {{ template "mariadb.chart" . }}

It will fail with:

Error: UPGRADE FAILED: StatefulSet.apps "redis-test-master" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.

Let's remove this chart label, uninstall and try to reproduce: the bug is gone.

Conclusion

• tl;dr: DO NOT put fields that may change in volumeClaimTemplate, especially chart label.
• Any fix to this is a breaking change. But any other unrelated change to already impacted chart breaks existing Releases.
• A potential workaround or "how to upgrade" is to remove the StatefulSet:

kubectl delete statefulsets.apps --cascade=false test-redis-master

and upgrade.

cc @mattfarina @scottrigby @juan131 @javsalgar @unguiculus @davidkarlsen @cpanato