Skip to content

Reconciliation failing after upgrade to 2.2.2 #4529

@Djiit

Description

@Djiit

Describe the bug

When the kustomize-controller reconcile resources, it throws these logs:

  • info: server-side apply completed (with all resources unchanged -- this is expected)
  • error : health check failed after 1.145123312s: failed early due to stalled resources: <resource> status: 'Unknown' (listing all the managed resources)

Steps to reproduce

Well, I'm not sure. It just started to happen yesterday, roughly a day after a 2.1.0 => 2.2.0 upgrade. Looks like one of my image was updated through an updatepolicy, then the reconciliation failed, and then every minute the kustomize-controller threw the logs above.

Expected behavior

Everything works (actually it works, but the controller is telling me ALL the resources are in a status "unknown"). I'd had to mute my notifications for now.

Screenshots and recordings

No response

OS / Distro

macOS latest version

Flux version

flux: v2.2.2

Flux check

❯ flux check
► checking prerequisites
✔ Kubernetes 1.26.11 >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.2.2
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.37.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.2.3
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.2.3
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.37.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.31.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.2.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

Github

Container Registry provider

No response

Additional context

Is it possible there is a threshold of some sort that was misconfigured when upgrading to 2.2.2 ?

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions