chore: reduce verbose logs and unnecessary reconcile #1312

zhuangqh · 2025-07-25T07:13:23Z

reduce verbose logs from knative

kaito-pr-agent · 2025-07-25T07:13:47Z

Title

Enhance Workspace updates with retry and reduce verbose logs

Description

Added retry mechanism for updating Workspace objects
Reduced verbose logs and unnecessary reconciliation
Added Knative logging configuration for ragengine and workspace charts

Changes walkthrough 📝

Relevant files

Enhancement

6 files

workspace_controller.go `Added updateWorkspaceWithRetry function`	+19/-2
workspace_gc_finalizer.go `Used updateWorkspaceWithRetry for finalizer removal`	+12/-6
knative-logging-configmap.yaml `Added knative logging configmap`	+35/-0
values.yaml `Added logging level configuration`	+3/-0
knative-logging-configmap.yaml `Added knative logging configmap`	+35/-0
values.yaml `Added logging level configuration`	+3/-0

Bug fix

1 files

workspace_status.go `Included IsConflict in retry conditions`	+1/-1

Need help?
Type /help how to ... in the comments thread for any questions about PR-Agent usage.
Check out the documentation for more information.

kaito-pr-agent · 2025-07-25T07:14:05Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Error Handling The error handling in `updateWorkspaceWithRetry` could be improved by checking for specific errors that should not trigger a retry, such as validation errors. return retry.RetryOnConflict(retry.DefaultRetry, func() error { latestWorkspace := &kaitov1beta1.Workspace{} if err := c.Get(ctx, client.ObjectKeyFromObject(wObj), latestWorkspace); err != nil { return err } if err := modifyFn(latestWorkspace); err != nil { return err } return c.Update(ctx, latestWorkspace) }) Logging Level The default logging level is set to "error". Consider providing more granular options or documentation on how to adjust this setting based on deployment needs. apiVersion: v1 kind: ConfigMap metadata: name: config-logging namespace: {{ .Release.Namespace }} data: loglevel.controller: {{ .Values.logging.level \| quote }} loglevel.webhook: {{ .Values.logging.level \| quote }} loglevel.autoscaler: {{ .Values.logging.level \| quote }} loglevel.queueproxy: {{ .Values.logging.level \| quote }} loglevel.activator: {{ .Values.logging.level \| quote }} loglevel.hpaautoscaler: {{ .Values.logging.level \| quote }} loglevel.certcontroller: {{ .Values.logging.level \| quote }} loglevel.istiocontroller: {{ .Values.logging.level \| quote }} zap-logger-config: \| { "level": {{ .Values.logging.level \| quote }}, "development": false, "outputPaths": ["stdout"], "errorOutputPaths": ["stderr"], "encoding": "json", "encoderConfig": { "timeKey": "ts", "levelKey": "level", "nameKey": "logger", "callerKey": "caller", "messageKey": "msg", "stacktraceKey": "stacktrace", "lineEnding": "", "levelEncoder": "", "timeEncoder": "iso8601", "durationEncoder": "", "callerEncoder": "" } } Logging Level The default logging level is set to "error". Consider providing more granular options or documentation on how to adjust this setting based on deployment needs. apiVersion: v1 kind: ConfigMap metadata: name: config-logging namespace: {{ .Release.Namespace }} data: loglevel.controller: {{ .Values.logging.level \| quote }} loglevel.webhook: {{ .Values.logging.level \| quote }} loglevel.autoscaler: {{ .Values.logging.level \| quote }} loglevel.queueproxy: {{ .Values.logging.level \| quote }} loglevel.activator: {{ .Values.logging.level \| quote }} loglevel.hpaautoscaler: {{ .Values.logging.level \| quote }} loglevel.certcontroller: {{ .Values.logging.level \| quote }} loglevel.istiocontroller: {{ .Values.logging.level \| quote }} zap-logger-config: \| { "level": {{ .Values.logging.level \| quote }}, "development": false, "outputPaths": ["stdout"], "errorOutputPaths": ["stderr"], "encoding": "json", "encoderConfig": { "timeKey": "ts", "levelKey": "level", "nameKey": "logger", "callerKey": "caller", "messageKey": "msg", "stacktraceKey": "stacktrace", "lineEnding": "", "levelEncoder": "", "timeEncoder": "iso8601", "durationEncoder": "", "callerEncoder": "" } }

kaito-pr-agent · 2025-07-25T07:15:00Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category

Suggestion

Impact

General

Move finalizer log outside error block

Remove the finalizer log message outside the error handling block to ensure it
always runs.

pkg/workspace/controllers/workspace_gc_finalizer.go [51-64]

 updateErr := updateWorkspaceWithRetry(ctx, c.Client, wObj, func(ws *kaitov1beta1.Workspace) error {
     controllerutil.RemoveFinalizer(ws, consts.WorkspaceFinalizer)
     return nil
 })
 if updateErr != nil {
     if apierrors.IsNotFound(updateErr) {
         return ctrl.Result{}, nil
     }
     klog.ErrorS(updateErr, "failed to update the workspace to remove finalizer", "workspace", klog.KObj(wObj))
     return ctrl.Result{}, updateErr
 }
+klog.InfoS("successfully removed the workspace finalizers", "workspace", klog.KObj(wObj))

Suggestion importance[1-10]: 3

__

Why: Moving the log message outside the error block ensures it always runs, but this is a minor improvement that does not significantly impact the functionality or correctness of the code.

Low

zhuangqh · 2025-07-25T07:18:34Z

knative logs per line
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.929774616Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:"POST", URL:(*url.URL)(0xc000b5d4d0), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Accept":[]string{"application/json, /"}, "Accept-Encoding":[]string{"gzip"}, "Content-Length":[]string{"5334"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"kube-apiserver-admission"}}, Body:(*http.body)(0xc000e4a640), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:5334, TransferEncoding:[]string(nil), Close:false, Host:"kaito-workspace-svc.kaito-workspace.svc:9443", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"10.244.8.10:40954", RequestURI:"/validate/workspace.kaito.sh?timeout=10s", TLS:(*tls.ConnectionState)(0xc00066e600), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:"/validate/workspace.kaito.sh", ctx:(*context.cancelCtx)(0xc00095ed20), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}

before

{"severity":"INFO","timestamp":"2025-07-25T00:34:42.801272758Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000de6bd0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"1998\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc00098e340), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:1998, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.18:60044\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066e000), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000933e50), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:34:42.801692       1 workspace_validation.go:62] "Validate creation" workspace="default/workspace-phi-4-mini"
I0725 00:34:42.801752       1 inference_config_validation.go:51] Inference config not specified. Using default: "inference-params-template"
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.801940071Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"CREATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"","knative.dev/userinfo":"masterclient","admissionreview/uid":"6981ab0c-4a17-4750-a27c-a09f75ee3636","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:34:42.817485       1 workspace_controller.go:98] "Reconciling" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.829704402Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000de7290), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"4045\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc00098ef40), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:4045, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.18:60044\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066e000), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc00095e1e0), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:34:42.830250       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.830299513Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"b63902f7-50d1-4d4d-b74f-fc648c5c737e","admissionreview/allowed":true,"admissionreview/result":"nil"}
2025-07-25T00:34:42Z	INFO	metadata.finalizers: "workspace.finalizer.kaito.sh": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers	{"controller": "workspace", "controllerGroup": "kaito.sh", "controllerKind": "Workspace", "Workspace": {"name":"workspace-phi-4-mini","namespace":"default"}, "namespace": "default", "name": "workspace-phi-4-mini", "reconcileID": "f837eea5-88e1-48bd-9a87-291b8cdc524f"}
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.879978964Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000b5c630), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"4592\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000dcd700), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:4592, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.18:60044\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066e000), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc00095e7d0), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:34:42.884251       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.884300046Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"92b05a54-b436-4f9a-8a2a-b7affd71909c","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:34:42.897505       1 workspace_controller.go:426] "no current nodes match the workspace resource spec" workspace="default/workspace-phi-4-mini"
I0725 00:34:42.897539       1 workspace_controller.go:358] "need to create more nodes" NodeCount=1
I0725 00:34:42.897562       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="NodeClaimReady" status="Unknown" reason="CreateNodeClaimPending" message="creating 1 nodeClaims"
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.929774616Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000b5d4d0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"5334\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000e4a640), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:5334, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.10:40954\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066e600), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc00095ed20), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:34:42.931351       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.931495749Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"status","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"90102d85-3ca1-4db5-9dd5-b70c55232838","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:34:42.943458       1 workspace_controller.go:480] "Creating multiple NodeClaims" count=1 workspace="default/workspace-phi-4-mini"
I0725 00:34:42.943496       1 controller.go:164] "Expecting creations" logger="WorkspaceController" controller="default/workspace-phi-4-mini" adds=1
I0725 00:34:42.944562       1 nodeclaim.go:89] "GenerateNodeClaimManifest" object={"kind":"Workspace","apiVersion":"kaito.sh/v1beta1","metadata":{"name":"workspace-phi-4-mini","namespace":"default","uid":"71dc10de-a2dd-4966-8fd6-da8d6552892c","resourceVersion":"122671097","generation":2,"creationTimestamp":"2025-07-25T00:34:42Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"kaito.sh/v1beta1\",\"inference\":{\"preset\":{\"name\":\"phi-4-mini-instruct\"}},\"kind\":\"Workspace\",\"metadata\":{\"annotations\":{},\"name\":\"workspace-phi-4-mini\",\"namespace\":\"default\"},\"resource\":{\"instanceType\":\"Standard_NC24ads_A100_v4\",\"labelSelector\":{\"matchLabels\":{\"apps\":\"phi-4\"}}}}\n","workspace.kaito.io/hash":"203bd2f573e212877fc413be56776945a635c4cd6ea73b05061e71a0272019a8","workspace.kaito.io/revision":"1"},"finalizers":["workspace.finalizer.kaito.sh"],"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"kaito.sh/v1beta1","time":"2025-07-25T00:34:42Z","fieldsType":"FieldsV1","fieldsV1":{"f:inference":{".":{},"f:preset":{".":{},"f:accessMode":{},"f:name":{}}},"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:resource":{".":{},"f:count":{},"f:instanceType":{},"f:labelSelector":{}}}},{"manager":"manager","operation":"Update","apiVersion":"kaito.sh/v1beta1","time":"2025-07-25T00:34:42Z","fieldsType":"FieldsV1","fieldsV1":{"f:inference":{"f:preset":{"f:presetOptions":{}}},"f:metadata":{"f:annotations":{"f:workspace.kaito.io/hash":{},"f:workspace.kaito.io/revision":{}},"f:finalizers":{".":{},"v:\"workspace.finalizer.kaito.sh\"":{}}}}}]},"resource":{"count":1,"instanceType":"Standard_NC24ads_A100_v4","labelSelector":{"matchLabels":{"apps":"phi-4"}}},"inference":{"preset":{"name":"phi-4-mini-instruct","accessMode":"public","presetOptions":{}}},"status":{}}
I0725 00:34:42.944626       1 nodeclaim.go:257] "CreateNodeClaim" nodeClaim="default/wse0666d818"
I0725 00:34:42.969271       1 workspace_event_handler.go:59] FUCK observe creation default/workspace-phi-4-mini
I0725 00:34:42.969572       1 controller.go:193] "FUCK Creation observed" logger="WorkspaceController" controller="default/workspace-phi-4-mini"
I0725 00:34:42.970016       1 workspace_controller.go:510] "NodeClaim created successfully" nodeClaim="wse0666d818" workspace="default/workspace-phi-4-mini"
I0725 00:34:42.970136       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="ResourceReady" status="False" reason="workspaceResourceCreated" message="nodeclaims created"
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.976130003Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000b6f3b0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"5919\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000e4bb00), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:5919, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.10:40954\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066e600), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc00095f540), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:34:42.977604       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:34:42.977645132Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"status","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"201831e1-0453-4412-b2e5-7fb52e04eed7","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:34:42.989074       1 workspace_controller.go:98] "Reconciling" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:34:43.000910777Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000cb4cf0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"6042\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000e7ab00), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6042, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.18:60044\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066e000), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc00095f9f0), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:34:43.002594       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:34:43.003642529Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"18fb70f4-2868-42d7-9869-eabf9cebd737","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:34:43.017593       1 nodeclaim.go:361] "CheckNodeClaimStatus" nodeClaim="wse0666d818"
I0725 00:38:43.156680       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="WorkspaceSucceeded" status="False" reason="workspaceFailed" message="check nodeClaim status timed out. nodeClaim wse0666d818 is not ready"
{"severity":"INFO","timestamp":"2025-07-25T00:38:43.191621672Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000a81830), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"6317\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000d93dc0), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6317, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.10:52596\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066e0c0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000b2f720), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:38:43.193407       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:38:43.193454913Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"status","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"a6656492-1e33-48dc-8755-d8e62bfbe7da","admissionreview/allowed":true,"admissionreview/result":"nil"}
2025-07-25T00:38:43Z	ERROR	Reconciler error	{"controller": "workspace", "controllerGroup": "kaito.sh", "controllerKind": "Workspace", "Workspace": {"name":"workspace-phi-4-mini","namespace":"default"}, "namespace": "default", "name": "workspace-phi-4-mini", "reconcileID": "8c90f7c5-ad53-4c12-b29d-18a099f0cf51", "error": "check nodeClaim status timed out. nodeClaim wse0666d818 is not ready"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202
I0725 00:38:43.205054       1 workspace_controller.go:98] "Reconciling" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:38:43.243674427Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000c4d200), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"6486\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000df7000), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6486, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.10:52608\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066e780), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000b2fe00), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:38:43.245775       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:38:43.245819575Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"c98bd477-baff-4f5e-844f-743b2f21816a","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:38:43.258070       1 nodeclaim.go:361] "CheckNodeClaimStatus" nodeClaim="wse0666d818"
I0725 00:40:32.315631       1 nodeclaim.go:400] "nodeClaim status is ready" nodeClaim="wse0666d818"
I0725 00:40:32.317869       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:33.320184       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:34.321022       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:35.321936       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:36.322900       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:37.323242       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:38.323911       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:39.324154       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:40.325258       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:41.332406       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-wse0666d818-32913337-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 00:40:42.332746       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="NodeClaimReady" status="True" reason="installNodePluginsSuccess" message="nodeClaim plugins have been installed successfully"
{"severity":"INFO","timestamp":"2025-07-25T00:40:42.365784624Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000db0630), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"6568\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc00092e7c0), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6568, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.10:45420\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066f800), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000c8adc0), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:40:42.367525       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:40:42.367568351Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"status","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"c23e42bf-e805-4312-a7bc-d8a22e41f377","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:40:42.381102       1 workspace_status.go:85] "updateStatusNodeList" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:40:42.417000918Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000db1ef0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"6670\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc00092f600), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6670, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.10:45422\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066fec0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000c8b310), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:40:42.418063       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:40:42.418132736Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"status","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"28ff8a1a-266c-4e6f-a82b-7a9eae22838e","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:40:42.434983       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="ResourceReady" status="True" reason="workspaceResourceStatusSuccess" message="workspace resource is ready"
{"severity":"INFO","timestamp":"2025-07-25T00:40:42.441664501Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000f4f950), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"6757\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000f34100), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6757, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.10:45420\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc00066f800), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000c8b590), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:40:42.442509       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:40:42.442615215Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"status","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"6c012e62-8da4-40a4-af28-889803bf2ffa","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:40:42.780180       1 resources.go:156] Default ConfigMap already exists in target namespace: default, no action taken.
I0725 00:40:42.780281       1 resources.go:39] "CreateService" service="default/workspace-phi-4-mini"
I0725 00:40:42.863585       1 resources.go:156] Default ConfigMap already exists in target namespace: default, no action taken.
I0725 00:40:43.203136       1 resources.go:35] "CreateDeployment" deployment="default/workspace-phi-4-mini"
I0725 00:44:04.234189       1 resources.go:96] "deployment status is ready" deployment="workspace-phi-4-mini"
I0725 00:44:04.234303       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="InferenceReady" status="True" reason="WorkspaceInferenceStatusSuccess" message="Inference has been deployed successfully"
{"severity":"INFO","timestamp":"2025-07-25T00:44:04.274679655Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000a81950), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"6976\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000d93000), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:6976, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.10:51472\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc0000e98c0), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000c3c5a0), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:44:04.276838       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:44:04.276885672Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"status","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"22bec914-80f9-4b7f-9ad6-05e87601116b","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:44:04.289935       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="WorkspaceSucceeded" status="True" reason="workspaceSucceeded" message="workspace succeeds"
{"severity":"INFO","timestamp":"2025-07-25T00:44:04.332191913Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000db0f30), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"7133\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000d93e80), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:7133, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.18:52254\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc000654480), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000c3c9b0), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:44:04.334096       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:44:04.334137228Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"status","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"24a042bb-ae60-4497-93ec-f2e095767947","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:44:04.347249       1 workspace_controller.go:98] "Reconciling" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:44:04.364171668Z","logger":"webhook","caller":"webhook/admission.go:93","message":"Webhook ServeHTTP request=&http.Request{Method:\"POST\", URL:(*url.URL)(0xc000f4ecf0), Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Accept\":[]string{\"application/json, */*\"}, \"Accept-Encoding\":[]string{\"gzip\"}, \"Content-Length\":[]string{\"7032\"}, \"Content-Type\":[]string{\"application/json\"}, \"User-Agent\":[]string{\"kube-apiserver-admission\"}}, Body:(*http.body)(0xc000df6ec0), GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:7032, TransferEncoding:[]string(nil), Close:false, Host:\"kaito-workspace-svc.kaito-workspace.svc:9443\", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:\"10.244.8.18:52254\", RequestURI:\"/validate/workspace.kaito.sh?timeout=10s\", TLS:(*tls.ConnectionState)(0xc000654480), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), Pattern:\"/validate/workspace.kaito.sh\", ctx:(*context.cancelCtx)(0xc000c3cf50), pat:(*http.pattern)(0xc000778de0), matches:[]string(nil), otherValues:map[string]string(nil)}"}
I0725 00:44:04.365943       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
{"severity":"INFO","timestamp":"2025-07-25T00:44:04.366026982Z","logger":"webhook","caller":"webhook/admission.go:151","message":"remote admission controller audit annotations=map[string]string(nil)","knative.dev/kind":"kaito.sh/v1beta1, Kind=Workspace","knative.dev/namespace":"default","knative.dev/name":"workspace-phi-4-mini","knative.dev/operation":"UPDATE","knative.dev/resource":"kaito.sh/v1beta1, Resource=workspaces","knative.dev/subresource":"","knative.dev/userinfo":"system:serviceaccount:kaito-workspace:kaito-workspace-sa","admissionreview/uid":"a452b749-5e68-4894-a61f-381a0916194d","admissionreview/allowed":true,"admissionreview/result":"nil"}
I0725 00:44:04.378871       1 resources.go:156] Default ConfigMap already exists in target namespace: default, no action taken.
I0725 00:44:04.379901       1 workspace_controller.go:741] "An inference workload already exists for workspace" workspace="default/workspace-phi-4-mini"

after

I0725 06:53:23.768185       1 workspace_validation.go:62] "Validate creation" workspace="default/workspace-phi-4-mini"
I0725 06:53:23.768446       1 inference_config_validation.go:51] Inference config not specified. Using default: "inference-params-template"
I0725 06:53:23.879987       1 workspace_controller.go:98] "Reconciling" workspace="default/workspace-phi-4-mini"
I0725 06:53:23.891945       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
2025-07-25T06:53:23Z	INFO	metadata.finalizers: "workspace.finalizer.kaito.sh": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers	{"controller": "workspace", "controllerGroup": "kaito.sh", "controllerKind": "Workspace", "Workspace": {"name":"workspace-phi-4-mini","namespace":"default"}, "namespace": "default", "name": "workspace-phi-4-mini", "reconcileID": "79961061-8e40-4ada-94d9-177487390397"}
I0725 06:53:23.952260       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 06:53:24.084774       1 workspace_controller.go:431] "no current nodes match the workspace resource spec" workspace="default/workspace-phi-4-mini"
I0725 06:53:24.084840       1 workspace_controller.go:363] "need to create more nodes" NodeCount=1
I0725 06:53:24.084868       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="NodeClaimReady" status="Unknown" reason="CreateNodeClaimPending" message="creating 1 nodeClaims"
I0725 06:53:24.146845       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 06:53:24.161244       1 workspace_controller.go:485] "Creating multiple NodeClaims" count=1 workspace="default/workspace-phi-4-mini"
I0725 06:53:24.161497       1 nodeclaim.go:89] "GenerateNodeClaimManifest" object={"kind":"Workspace","apiVersion":"kaito.sh/v1beta1","metadata":{"name":"workspace-phi-4-mini","namespace":"default","uid":"e459ffe1-989d-4b87-a764-72f0990e83ed","resourceVersion":"122784293","generation":1,"creationTimestamp":"2025-07-25T06:53:23Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"kaito.sh/v1beta1\",\"inference\":{\"preset\":{\"name\":\"phi-4-mini-instruct\"}},\"kind\":\"Workspace\",\"metadata\":{\"annotations\":{},\"name\":\"workspace-phi-4-mini\",\"namespace\":\"default\"},\"resource\":{\"instanceType\":\"Standard_NC24ads_A100_v4\",\"labelSelector\":{\"matchLabels\":{\"apps\":\"phi-4\"}}}}\n","workspace.kaito.io/hash":"203bd2f573e212877fc413be56776945a635c4cd6ea73b05061e71a0272019a8","workspace.kaito.io/revision":"1"},"finalizers":["workspace.finalizer.kaito.sh"],"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"kaito.sh/v1beta1","time":"2025-07-25T06:53:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:inference":{".":{},"f:preset":{".":{},"f:accessMode":{},"f:name":{}}},"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:resource":{".":{},"f:count":{},"f:instanceType":{},"f:labelSelector":{}}}},{"manager":"manager","operation":"Update","apiVersion":"kaito.sh/v1beta1","time":"2025-07-25T06:53:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:finalizers":{".":{},"v:\"workspace.finalizer.kaito.sh\"":{}}}}}]},"resource":{"count":1,"instanceType":"Standard_NC24ads_A100_v4","labelSelector":{"matchLabels":{"apps":"phi-4"}}},"inference":{"preset":{"name":"phi-4-mini-instruct","accessMode":"public","presetOptions":{}}},"status":{}}
I0725 06:53:24.161627       1 nodeclaim.go:257] "CreateNodeClaim" nodeClaim="default/ws0edba3f4c"
I0725 06:53:24.185223       1 workspace_controller.go:515] "NodeClaim created successfully" nodeClaim="ws0edba3f4c" workspace="default/workspace-phi-4-mini"
I0725 06:53:24.185265       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="ResourceReady" status="False" reason="workspaceResourceCreated" message="nodeclaims created successfully"
I0725 06:53:24.194723       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 06:53:24.207735       1 workspace_controller.go:98] "Reconciling" workspace="default/workspace-phi-4-mini"
I0725 06:53:24.224977       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 06:53:24.247638       1 nodeclaim.go:361] "CheckNodeClaimStatus" nodeClaim="ws0edba3f4c"
I0725 06:57:24.379406       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="WorkspaceSucceeded" status="False" reason="workspaceFailed" message="check nodeClaim status timed out. nodeClaim ws0edba3f4c is not ready"
I0725 06:57:24.422455       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
2025-07-25T06:57:24Z	ERROR	Reconciler error	{"controller": "workspace", "controllerGroup": "kaito.sh", "controllerKind": "Workspace", "Workspace": {"name":"workspace-phi-4-mini","namespace":"default"}, "namespace": "default", "name": "workspace-phi-4-mini", "reconcileID": "0c42e2bb-233c-4917-948c-5da26d368475", "error": "check nodeClaim status timed out. nodeClaim ws0edba3f4c is not ready"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202
I0725 06:57:24.443357       1 workspace_controller.go:98] "Reconciling" workspace="default/workspace-phi-4-mini"
I0725 06:57:24.501325       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 06:57:24.516140       1 nodeclaim.go:361] "CheckNodeClaimStatus" nodeClaim="ws0edba3f4c"
I0725 06:57:38.525060       1 nodeclaim.go:400] "nodeClaim status is ready" nodeClaim="ws0edba3f4c"
I0725 06:57:38.526060       1 nodes.go:61] "UpdateNodeWithLabel" nodeName="aks-ws0edba3f4c-37942892-vmss000000" labelKey="accelerator" labelValue="nvidia"
I0725 06:58:06.545391       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="NodeClaimReady" status="True" reason="installNodePluginsSuccess" message="nodeClaim plugins have been installed successfully"
I0725 06:58:06.561301       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 06:58:06.582900       1 workspace_status.go:85] "updateStatusNodeList" workspace="default/workspace-phi-4-mini"
I0725 06:58:06.590716       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 06:58:06.611806       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="ResourceReady" status="True" reason="workspaceResourceStatusSuccess" message="workspace resource is ready"
I0725 06:58:06.629004       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 06:58:06.967341       1 resources.go:156] Default ConfigMap already exists in target namespace: default, no action taken.
I0725 06:58:06.967463       1 resources.go:39] "CreateService" service="default/workspace-phi-4-mini"
I0725 06:58:06.999174       1 resources.go:156] Default ConfigMap already exists in target namespace: default, no action taken.
I0725 06:58:07.320040       1 resources.go:35] "CreateDeployment" deployment="default/workspace-phi-4-mini"
I0725 07:01:08.360603       1 resources.go:96] "deployment status is ready" deployment="workspace-phi-4-mini"
I0725 07:01:08.360958       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="InferenceReady" status="True" reason="WorkspaceInferenceStatusSuccess" message="Inference has been deployed successfully"
I0725 07:01:08.401445       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 07:01:08.415884       1 workspace_status.go:65] "updateStatusCondition" workspace="default/workspace-phi-4-mini" conditionType="WorkspaceSucceeded" status="True" reason="workspaceSucceeded" message="workspace succeeds"
I0725 07:01:08.481942       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 07:01:08.494942       1 workspace_controller.go:98] "Reconciling" workspace="default/workspace-phi-4-mini"
I0725 07:01:08.510957       1 workspace_validation.go:87] "Validate update" workspace="default/workspace-phi-4-mini"
I0725 07:01:08.525905       1 resources.go:156] Default ConfigMap already exists in target namespace: default, no action taken.
I0725 07:01:08.526206       1 workspace_controller.go:746] "An inference workload already exists for workspace" workspace="default/workspace-phi-4-mini"

Signed-off-by: zhuangqh <zhuangqhc@gmail.com>

zhuangqh requested review from Fei-Guo and chewong as code owners July 25, 2025 07:13

github-project-automation bot added this to KAITO Roadmap Jul 25, 2025

zhuangqh had a problem deploying to unit-tests July 25, 2025 07:13 — with GitHub Actions Failure

zhuangqh had a problem deploying to e2e-test July 25, 2025 07:13 — with GitHub Actions Error

zhuangqh temporarily deployed to unit-tests July 25, 2025 07:13 — with GitHub Actions Inactive

kaito-pr-agent bot added the Review effort 4/5 label Jul 25, 2025

chore: reduce verbose logs and unnecessary reconcile

b18de74

Signed-off-by: zhuangqh <zhuangqhc@gmail.com>

zhuangqh force-pushed the fix-log branch from 88a6747 to b18de74 Compare July 25, 2025 08:24

zhuangqh temporarily deployed to unit-tests July 25, 2025 08:24 — with GitHub Actions Inactive

zhuangqh temporarily deployed to e2e-test July 25, 2025 08:24 — with GitHub Actions Inactive

Merge branch 'main' into fix-log

94158c1

Fei-Guo temporarily deployed to unit-tests July 28, 2025 05:44 — with GitHub Actions Inactive

Fei-Guo requested a deployment to e2e-test July 28, 2025 05:44 — with GitHub Actions Waiting

Fei-Guo approved these changes Jul 28, 2025

View reviewed changes

Fei-Guo merged commit 8c7ac49 into kaito-project:main Jul 28, 2025
11 of 12 checks passed

github-project-automation bot moved this to Done in KAITO Roadmap Jul 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: reduce verbose logs and unnecessary reconcile #1312

chore: reduce verbose logs and unnecessary reconcile #1312

Uh oh!

zhuangqh commented Jul 25, 2025 •

edited

Loading

Uh oh!

kaito-pr-agent bot commented Jul 25, 2025

Uh oh!

kaito-pr-agent bot commented Jul 25, 2025

Uh oh!

kaito-pr-agent bot commented Jul 25, 2025

Uh oh!

zhuangqh commented Jul 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

chore: reduce verbose logs and unnecessary reconcile #1312

chore: reduce verbose logs and unnecessary reconcile #1312

Uh oh!

Conversation

zhuangqh commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaito-pr-agent bot commented Jul 25, 2025

Title

Description

Changes walkthrough 📝

Uh oh!

kaito-pr-agent bot commented Jul 25, 2025

PR Reviewer Guide 🔍

Uh oh!

kaito-pr-agent bot commented Jul 25, 2025

PR Code Suggestions ✨

Uh oh!

zhuangqh commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhuangqh commented Jul 25, 2025 •

edited

Loading

zhuangqh commented Jul 25, 2025 •

edited

Loading