Skip to content

CI: RuntimeKVStoreTest: Stopping cilium during Runtime tests #11895

@pchaigno

Description

@pchaigno

RuntimeKVStoreTest Consul KVStore and RuntimeKVStoreTest Etcd KVStore are failing often with different error messages. The root cause, found in the logs, seems to be:

Jun 03 01:40:59 runtime systemd[1]: Stopping cilium...

RuntimeKVStoreTest Consul KVStore usually fails with:

home/jenkins/workspace/Ginkgo-CI-Tests-runtime-Pipeline/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:421
Found 2 Cilium logs matching list of errors that must be investigated
/home/jenkins/workspace/Ginkgo-CI-Tests-runtime-Pipeline/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:419
Standard output
⚠️  Found a "JoinEP: " in logs
⚠️  Found a "JoinEP: " in logs

because the BPF compilation gets interrupted by Cilium stopping.
https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/4989/testReport/junit/(root)/Suite-runtime/RuntimeKVStoreTest_Etcd_KVStore/
https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/4908/testReport/junit/(root)/Suite-runtime/RuntimeKVStoreTest_Etcd_KVStore/
15d82b0c_RuntimeKVStoreTest_Etcd_KVStore.zip
2533d569_RuntimeKVStoreTest_Etcd_KVStore.zip


RuntimeKVStoreTest Etcd KVStore usually fails with:

Error getting names of endpoints from cilium
Expected
    <*errors.errorString | 0xc0011ce610>: {
        s: "`cilium endpoint list` was not successful",
    }
to be nil

because Cilium is not running anymore.
https://jenkins.cilium.io/job/Ginkgo-CI-Tests-runtime-Pipeline/416/testReport/junit/(root)/Suite-runtime/RuntimeKVStoreTest_Consul_KVStore/
https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/4989/testReport/junit/(root)/Suite-runtime/RuntimeKVStoreTest_Consul_KVStore/
test_results_Ginkgo-CI-Tests-runtime-Pipeline_416_BDD-Test-PR.zip
test_results_master_4989_BDD-Test-PR-runtime.zip

Metadata

Metadata

Assignees

Labels

area/CIContinuous Integration testing issue or flakeci/flakeThis is a known failure that occurs in the tree. Please investigate me!

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions