Oscillating reconciles for aggregated clusterroles #1041
Description
After upgrading Flux from version 2.1.2 to 2.2.1 (kustomize-controller from version 1.1.1 to 1.2.1), we observe oscillating reconciles for all our aggregated clusterroles. I asked on Slack about this but haven't gotten anything back yet.
This happens for all our aggregated clusterroles originating from Kyverno and Crossplane Helm chart. Note: even if the resource examples indicate that we are using Helm, this is untrue. We inflate all Helm charts into YAML manifests using kustomize (CLI) and use a Flux Kustomization to provision. I will use one of the Kyverno aggregated clusterroles as an example, but the symptom is the same for all aggregated clusterroles in our Flux installation.
Expected behavior (Flux 2.1.2): When Flux reconciles a Kustomization containing an aggregated clusterrole (without any rules in source), the SSA should be a no-op and don't create any update event.
Actualt behavior (Flux 2.2.1): When Flux reconciles a Kustomization containing an aggregated clusterrole (without any rules in source), it seems like the SSA performed by Flux is resetting the rules field creating an update event for each aggregated clusterrole. The rules field is immediately populated by the aggregated clusterrole controller. So the next time Flux performs a reconcile, this happens again....
Example clusterrole in source:
aggregationRule:
clusterRoleSelectors:
- matchLabels:
app.kubernetes.io/component: reports-controller
app.kubernetes.io/instance: kyverno
app.kubernetes.io/part-of: kyverno
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: reports-controller
app.kubernetes.io/instance: kyverno
app.kubernetes.io/part-of: kyverno
app.kubernetes.io/version: 3.1.1
name: kyverno:reports-controllerResulting clusterrole in one of our clusters (note the timestamps on the managed fields)
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: 'kyverno:reports-controller'
uid: 435e6c86-269e-41a6-abc7-f621d70a0e3a
resourceVersion: '845187637'
creationTimestamp: '2023-08-10T11:22:40Z'
labels:
app.kubernetes.io/component: reports-controller
app.kubernetes.io/instance: kyverno
app.kubernetes.io/part-of: kyverno
app.kubernetes.io/version: 3.1.1
kustomize.toolkit.fluxcd.io/name: kyverno
kustomize.toolkit.fluxcd.io/namespace: flux-system
managedFields:
- manager: clusterrole-aggregation-controller
operation: Apply
apiVersion: rbac.authorization.k8s.io/v1
time: '2023-12-21T13:53:25Z'
fieldsType: FieldsV1
fieldsV1:
'f:rules': {}
- manager: kustomize-controller
operation: Apply
apiVersion: rbac.authorization.k8s.io/v1
time: '2023-12-21T13:53:25Z'
fieldsType: FieldsV1
fieldsV1:
'f:aggregationRule':
'f:clusterRoleSelectors': {}
'f:metadata':
'f:labels':
'f:app.kubernetes.io/component': {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/part-of': {}
'f:app.kubernetes.io/version': {}
'f:kustomize.toolkit.fluxcd.io/name': {}
'f:kustomize.toolkit.fluxcd.io/namespace': {}
rules:
- verbs:
- get
- list
- watch
apiGroups:
- '*'
resources:
- '*'
- verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- deletecollection
apiGroups:
- kyverno.io
resources:
- admissionreports
- clusteradmissionreports
- backgroundscanreports
- clusterbackgroundscanreports
- verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- deletecollection
apiGroups:
- wgpolicyk8s.io
resources:
- policyreports
- policyreports/status
- clusterpolicyreports
- clusterpolicyreports/status
- verbs:
- create
- patch
apiGroups:
- ''
- events.k8s.io
resources:
- events
aggregationRule:
clusterRoleSelectors:
- matchLabels:
app.kubernetes.io/component: reports-controller
app.kubernetes.io/instance: kyverno
app.kubernetes.io/part-of: kyverno