Skip to content

[v1.13,v1.12] CI: K8sDatapathConfig MonitorAggregation: Error from server: etcdserver: request timed out #24839

@gandro

Description

@gandro

Test Name

K8sDatapathConfig MonitorAggregation Checks that monitor aggregation restricts notifications

Failure Output

FAIL: Unable to retrieve DNS pods to scale down, command 'kubectl get deploy -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].status.replicas}'': Exitcode: 1

Stack Trace

/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Unable to retrieve DNS pods to scale down, command 'kubectl get deploy -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].status.replicas}'': Exitcode: 1 
Err: exit status 1
Stdout:
 	 
Stderr:
 	 Error from server: etcdserver: request timed out

Standard Output

Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 5
Number of "level=warning" in logs: 2
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
error retrieving resource lock kube-system/cilium-operator-resource-lock: Get \
Network status error received, restarting client connections
Failed to release lock: Put \
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
⚠️  Number of "level=warning" in logs: 8
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Waiting for k8s node information
Unable to get node resource
Network status error received, restarting client connections
Key allocation attempt failed
Cilium pods: [cilium-gs5cd cilium-pfzdj]
Netpols loaded: 
CiliumNetworkPolicies loaded: 
Endpoint Policy Enforcement:
Pod                           Ingress   Egress
prometheus-669755c8c5-xsnqq   false     false
coredns-bb76b858c-nj6dn       false     false
grafana-7ddfc74b5b-sp8f6      false     false
Cilium agent 'cilium-gs5cd': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 18 Failed 0
Cilium agent 'cilium-pfzdj': Status: Ok  Health: Ok Nodes "" ContainerRuntime:  Kubernetes: Ok KVstore: Ok Controllers: Total 30 Failed 0

Standard Error

17:54:43 STEP: Running BeforeAll block for EntireTestsuite K8sDatapathConfig
17:54:43 STEP: Ensuring the namespace kube-system exists
17:54:43 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
17:54:43 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
17:54:43 STEP: Installing Cilium
17:54:44 STEP: Waiting for Cilium to become ready
17:55:10 STEP: Restarting unmanaged pods coredns-bb76b858c-xmg4x in namespace kube-system
17:55:17 STEP: Validating if Kubernetes DNS is deployed
17:55:17 STEP: Checking if deployment is ready
17:55:17 STEP: Checking if kube-dns service is plumbed correctly
17:55:17 STEP: Checking if pods have identity
17:55:17 STEP: Checking if DNS can resolve
17:55:30 STEP: Kubernetes DNS is not ready: 5s timeout expired
17:55:30 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
FAIL: Unable to retrieve DNS pods to scale down, command 'kubectl get deploy -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].status.replicas}'': Exitcode: 1 
Err: exit status 1
Stdout:
 	 
Stderr:
 	 Error from server: etcdserver: request timed out
	 

=== Test Finished at 2023-04-11T17:55:38Z====
17:55:38 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
17:55:39 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0 
Stdout:
 	 NAMESPACE           NAME                              READY   STATUS              RESTARTS   AGE     IP              NODE   NOMINATED NODE   READINESS GATES
	 cilium-monitoring   grafana-7ddfc74b5b-sp8f6          0/1     Running             0          63s     10.0.0.128      k8s2   <none>           <none>
	 cilium-monitoring   prometheus-669755c8c5-xsnqq       0/1     ContainerCreating   0          63s     <none>          k8s2   <none>           <none>
	 kube-system         cilium-gs5cd                      1/1     Running             0          61s     192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-operator-878c95dbc-2ths2   1/1     Running             0          61s     192.168.56.12   k8s2   <none>           <none>
	 kube-system         cilium-operator-878c95dbc-62cm4   1/1     Running             1          61s     192.168.56.11   k8s1   <none>           <none>
	 kube-system         cilium-pfzdj                      1/1     Running             0          61s     192.168.56.12   k8s2   <none>           <none>
	 kube-system         coredns-bb76b858c-nj6dn           1/1     Running             0          35s     10.0.0.59       k8s2   <none>           <none>
	 kube-system         etcd-k8s1                         1/1     Running             0          5m34s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-apiserver-k8s1               1/1     Running             0          5m34s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-controller-manager-k8s1      0/1     CrashLoopBackOff    2          5m34s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-proxy-nmhvf                  1/1     Running             0          112s    192.168.56.12   k8s2   <none>           <none>
	 kube-system         kube-proxy-qlcqv                  1/1     Running             0          3m1s    192.168.56.11   k8s1   <none>           <none>
	 kube-system         kube-scheduler-k8s1               0/1     CrashLoopBackOff    2          5m34s   192.168.56.11   k8s1   <none>           <none>
	 kube-system         log-gatherer-fn8bg                1/1     Running             0          66s     192.168.56.12   k8s2   <none>           <none>
	 kube-system         log-gatherer-lvmck                1/1     Running             0          66s     192.168.56.11   k8s1   <none>           <none>
	 kube-system         registry-adder-c4dmf              1/1     Running             0          110s    192.168.56.12   k8s2   <none>           <none>
	 kube-system         registry-adder-qcr4w              1/1     Running             0          110s    192.168.56.11   k8s1   <none>           <none>
	 
Stderr:

Resources

Anything else?

Potentially related to #23892 (similar etc error)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/CIContinuous Integration testing issue or flakeci/flakeThis is a known failure that occurs in the tree. Please investigate me!staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions