-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Closed as not planned
Closed as not planned
Copy link
Labels
area/CIContinuous Integration testing issue or flakeContinuous Integration testing issue or flakeci/flakeThis is a known failure that occurs in the tree. Please investigate me!This is a known failure that occurs in the tree. Please investigate me!staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Description
Test Name
K8sDatapathConfig MonitorAggregation Checks that monitor aggregation restricts notifications
Failure Output
FAIL: Unable to retrieve DNS pods to scale down, command 'kubectl get deploy -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].status.replicas}'': Exitcode: 1
Stack Trace
/home/jenkins/workspace/Cilium-PR-K8s-1.20-kernel-4.9/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:515
Unable to retrieve DNS pods to scale down, command 'kubectl get deploy -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].status.replicas}'': Exitcode: 1
Err: exit status 1
Stdout:
Stderr:
Error from server: etcdserver: request timed out
Standard Output
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
No errors/warnings found in logs
Number of "context deadline exceeded" in logs: 2
Number of "level=error" in logs: 5
Number of "level=warning" in logs: 2
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 3 errors/warnings:
error retrieving resource lock kube-system/cilium-operator-resource-lock: Get \
Network status error received, restarting client connections
Failed to release lock: Put \
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
⚠️ Number of "level=warning" in logs: 8
Number of "Cilium API handler panicked" in logs: 0
Number of "Goroutine took lock for more than" in logs: 0
Top 4 errors/warnings:
Waiting for k8s node information
Unable to get node resource
Network status error received, restarting client connections
Key allocation attempt failed
Cilium pods: [cilium-gs5cd cilium-pfzdj]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
Pod Ingress Egress
prometheus-669755c8c5-xsnqq false false
coredns-bb76b858c-nj6dn false false
grafana-7ddfc74b5b-sp8f6 false false
Cilium agent 'cilium-gs5cd': Status: Ok Health: Ok Nodes "" ContainerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 18 Failed 0
Cilium agent 'cilium-pfzdj': Status: Ok Health: Ok Nodes "" ContainerRuntime: Kubernetes: Ok KVstore: Ok Controllers: Total 30 Failed 0
Standard Error
17:54:43 STEP: Running BeforeAll block for EntireTestsuite K8sDatapathConfig
17:54:43 STEP: Ensuring the namespace kube-system exists
17:54:43 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs")
17:54:43 STEP: WaitforPods(namespace="kube-system", filter="-l k8s-app=cilium-test-logs") => <nil>
17:54:43 STEP: Installing Cilium
17:54:44 STEP: Waiting for Cilium to become ready
17:55:10 STEP: Restarting unmanaged pods coredns-bb76b858c-xmg4x in namespace kube-system
17:55:17 STEP: Validating if Kubernetes DNS is deployed
17:55:17 STEP: Checking if deployment is ready
17:55:17 STEP: Checking if kube-dns service is plumbed correctly
17:55:17 STEP: Checking if pods have identity
17:55:17 STEP: Checking if DNS can resolve
17:55:30 STEP: Kubernetes DNS is not ready: 5s timeout expired
17:55:30 STEP: Restarting Kubernetes DNS (-l k8s-app=kube-dns)
FAIL: Unable to retrieve DNS pods to scale down, command 'kubectl get deploy -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].status.replicas}'': Exitcode: 1
Err: exit status 1
Stdout:
Stderr:
Error from server: etcdserver: request timed out
=== Test Finished at 2023-04-11T17:55:38Z====
17:55:38 STEP: Running JustAfterEach block for EntireTestsuite K8sDatapathConfig
===================== TEST FAILED =====================
17:55:39 STEP: Running AfterFailed block for EntireTestsuite K8sDatapathConfig
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cilium-monitoring grafana-7ddfc74b5b-sp8f6 0/1 Running 0 63s 10.0.0.128 k8s2 <none> <none>
cilium-monitoring prometheus-669755c8c5-xsnqq 0/1 ContainerCreating 0 63s <none> k8s2 <none> <none>
kube-system cilium-gs5cd 1/1 Running 0 61s 192.168.56.11 k8s1 <none> <none>
kube-system cilium-operator-878c95dbc-2ths2 1/1 Running 0 61s 192.168.56.12 k8s2 <none> <none>
kube-system cilium-operator-878c95dbc-62cm4 1/1 Running 1 61s 192.168.56.11 k8s1 <none> <none>
kube-system cilium-pfzdj 1/1 Running 0 61s 192.168.56.12 k8s2 <none> <none>
kube-system coredns-bb76b858c-nj6dn 1/1 Running 0 35s 10.0.0.59 k8s2 <none> <none>
kube-system etcd-k8s1 1/1 Running 0 5m34s 192.168.56.11 k8s1 <none> <none>
kube-system kube-apiserver-k8s1 1/1 Running 0 5m34s 192.168.56.11 k8s1 <none> <none>
kube-system kube-controller-manager-k8s1 0/1 CrashLoopBackOff 2 5m34s 192.168.56.11 k8s1 <none> <none>
kube-system kube-proxy-nmhvf 1/1 Running 0 112s 192.168.56.12 k8s2 <none> <none>
kube-system kube-proxy-qlcqv 1/1 Running 0 3m1s 192.168.56.11 k8s1 <none> <none>
kube-system kube-scheduler-k8s1 0/1 CrashLoopBackOff 2 5m34s 192.168.56.11 k8s1 <none> <none>
kube-system log-gatherer-fn8bg 1/1 Running 0 66s 192.168.56.12 k8s2 <none> <none>
kube-system log-gatherer-lvmck 1/1 Running 0 66s 192.168.56.11 k8s1 <none> <none>
kube-system registry-adder-c4dmf 1/1 Running 0 110s 192.168.56.12 k8s2 <none> <none>
kube-system registry-adder-qcr4w 1/1 Running 0 110s 192.168.56.11 k8s1 <none> <none>
Stderr:
Resources
- Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.9/1781/testReport/junit/Suite-k8s-1/20/K8sDatapathConfig_MonitorAggregation_Checks_that_monitor_aggregation_restricts_notifications/
- ZIP file(s):
499299b7_K8sDatapathConfig_MonitorAggregation_Checks_that_monitor_aggregation_restricts_notifications.zip
Anything else?
Potentially related to #23892 (similar etc error)
Metadata
Metadata
Assignees
Labels
area/CIContinuous Integration testing issue or flakeContinuous Integration testing issue or flakeci/flakeThis is a known failure that occurs in the tree. Please investigate me!This is a known failure that occurs in the tree. Please investigate me!staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.