-
Notifications
You must be signed in to change notification settings - Fork 151
Closed
Description
Just noticed that kamaji stopped processing new clusters, readiness probe reported ok, logs are looping over:
2025-01-28T19:19:59Z INFO soot_tenant-k8s2user19549_kubernetes-k8s2user19549.PhaseClusterAdminRBAC reconciliation completed
2025-01-28T19:20:31Z ERROR soot_tenant-k8suser16735_kubernetes-1k8s16735 kubeadm function failed {"controller": "clusterrolebinding", "controllerGroup": "rbac.authorization.k8s.io", "controllerKind": "ClusterRoleBinding", "ClusterRoleBinding": {"name":"kubeadm:get-nodes"}, "namespace": "", "name": "kubeadm:get-nodes", "reconcileID": "b7e33a3d-b36f-428b-94a8-05ba12a53383", "resource": "PhaseClusterAdminRBAC", "phase": "PhaseClusterAdminRBAC", "error": "unable to create ClusterRoleBinding: client rate limiter Wait returned an error: context deadline exceeded", "errorVerbose": "client rate limiter Wait returned an error: context deadline exceeded\nunable to create ClusterRoleBinding\nk8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.EnsureAdminClusterRoleBindingImpl.func1\n\t/go/pkg/mod/k8s.io/kubernetes@v1.31.1/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:714\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2\n\t/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/loop.go:87\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/loop.go:88\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextTimeout\n\t/go/pkg/mod/k8s.io/apimachinery@v0.31.1/pkg/util/wait/poll.go:48\nk8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.EnsureAdminClusterRoleBindingImpl\n\t/go/pkg/mod/k8s.io/kubernetes@v1.31.1/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:692\nk8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.EnsureAdminClusterRoleBinding\n\t/go/pkg/mod/k8s.io/kubernetes@v1.31.1/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:652\ngithub.com/clastix/kamaji/internal/resources.(*KubeadmPhase).GetKubeadmFunction.func2\n\t/workspace/internal/resources/kubeadm_phases.go:167\ngithub.com/clastix/kamaji/internal/resources.KubeadmPhaseCreate\n\t/workspace/internal/resources/kubeadm_utils.go:151\ngithub.com/clastix/kamaji/internal/resources.(*KubeadmPhase).CreateOrUpdate\n\t/workspace/internal/resources/kubeadm_phases.go:241\ngithub.com/clastix/kamaji/internal/resources.createOrUpdate\n\t/workspace/internal/resources/resource.go:92\ngithub.com/clastix/kamaji/internal/resources.Handle\n\t/workspace/internal/resources/resource.go:67\ngithub.com/clastix/kamaji/controllers/soot/controllers.(*KubeadmPhase).Reconcile\n\t/workspace/controllers/soot/controllers/kubeadm_phase.go:42\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/clastix/kamaji/internal/resources.KubeadmPhaseCreate
/workspace/internal/resources/kubeadm_utils.go:152
github.com/clastix/kamaji/internal/resources.(*KubeadmPhase).CreateOrUpdate
/workspace/internal/resources/kubeadm_phases.go:241
github.com/clastix/kamaji/internal/resources.createOrUpdate
/workspace/internal/resources/resource.go:92
github.com/clastix/kamaji/internal/resources.Handle
/workspace/internal/resources/resource.go:67
github.com/clastix/kamaji/controllers/soot/controllers.(*KubeadmPhase).Reconcile
/workspace/controllers/soot/controllers/kubeadm_phase.go:42
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:224
Metadata
Metadata
Assignees
Labels
No labels