Skip to content

CI: RuntimeConntrackInVethModeTest Conntrack-related configuration options for endpoints: curl from to httpd2 fails #19124

@joestringer

Description

@joestringer

Test Name

RuntimeConntrackInVethModeTest Conntrack-related configuration options for endpoints

Failure Output

FAIL: The result of "curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 http://[f00d::a0f:0:0:f43a]:80 -w \"time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'\"" from container "curl" to httpd2 does not match
Expected
    <bool>: true
to be false

Stack Trace

/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:527
The result of "curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 http://[f00d::a0f:0:0:f43a]:80 -w \"time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'\"" from container "curl" to httpd2 does not match
Expected
    <bool>: true
to be false
/home/jenkins/workspace/Cilium-PR-Runtime-net-next/runtime-gopath/src/github.com/cilium/cilium/test/runtime/connectivity.go:282

Standard Output

⚠️  Found "panic:" in logs 1 times
Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
⚠️  Number of "level=warning" in logs: 7
Number of "Cilium API handler panicked" in logs: 0
⚠️  Number of "Goroutine took lock for more than" in logs: 10
Top 2 errors/warnings:
Re-created missing slave key
Support for Consul as a kvstore backend has been deprecated due to lack of maintainers. If you are interested in helping to maintain Consul support in Cilium, please reach out on GitHub or the official Cilium slack

Standard Error

17:38:17 STEP: Running BeforeAll block for EntireTestsuite RuntimeConntrackInVethModeTest
17:38:19 STEP: Setting PolicyEnforcement=always
17:38:21 STEP: Running BeforeEach block for EntireTestsuite RuntimeConntrackInVethModeTest
17:38:29 STEP: Setting up policy: /home/vagrant/go/src/github.com/cilium/cilium/test/runtime/manifests/ct-test-policy.json
17:38:29 STEP: Getting Endpoint IDs
17:38:29 STEP: Endpoint ID for "server" = "1416"
17:38:29 STEP: Endpoint ID for "client" = "743"
17:38:29 STEP: Testing with endpoint configuration option: ConntrackLocal=Disabled
17:38:29 STEP: ============= Starting Connectivity Test ============= 
17:38:29 STEP: Getting IPs of each spawned container
17:38:29 STEP: client container Docker networking: map["EndpointID":"dd78d04953ceaf3e6862e7e74d9e58dd7787207443bd02b79b6a340bf30f4eca" "IPv4":"10.15.3.86" "IPv6":"f00d::a0f:0:0:22e3" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"client" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:38:29 STEP: server container Docker networking: map["EndpointID":"7547a804e1172e1e103ad6cbeb9c55957e8dbad6e3350d2104b69ab970aae2b0" "IPv4":"10.15.147.41" "IPv6":"f00d::a0f:0:0:9b29" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"server" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:38:30 STEP: httpd1 container Docker networking: map["EndpointID":"54d2e0a87a5c1f91c254d1bb7022b506f0ed3d51277b6eb605b8576606478945" "IPv4":"10.15.63.56" "IPv6":"f00d::a0f:0:0:e2b4" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"httpd1" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:38:30 STEP: httpd2 container Docker networking: map["EndpointID":"bab6b0bad18a4e77baa156fe8dee345527e592b7e1d51f4c40c6029adc8af99f" "IPv4":"10.15.60.130" "IPv6":"f00d::a0f:0:0:f43a" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"httpd2" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:38:30 STEP: curl1 container Docker networking: map["EndpointID":"f8376f92c5ce585d315dcf061243f839f1743b5d0bd41d9e1b3e06c56d4f7401" "IPv4":"10.15.173.123" "IPv6":"f00d::a0f:0:0:c6c" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"curl" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:38:30 STEP: httpd1 container Docker networking: map["EndpointID":"b265856d5a542524ea13f60f093740274b7ff23b3123def85bbeea81601f8237" "IPv4":"10.15.158.213" "IPv6":"f00d::a0f:0:0:4d9a" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"curl2" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:38:30 STEP: Showing policies imported to Cilium
[
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.curl": ""
      }
    },
    "egress": [
      {
        "toPorts": [
          {
            "ports": [
              {
                "port": "80",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "curl",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.server": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "reserved:host": ""
            }
          },
          {
            "matchLabels": {
              "any:id.client": ""
            }
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "server",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.client": ""
      }
    },
    "egress": [
      {
        "toEndpoints": [
          {
            "matchLabels": {
              "any:id.server": ""
            }
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "egress-from-client-to-server",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.httpd": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "any:id.curl": ""
            }
          }
        ],
        "toPorts": [
          {
            "ports": [
              {
                "port": "80",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "httpd",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.httpd": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "any:id.curl2": ""
            }
          }
        ],
        "toPorts": [
          {
            "ports": [
              {
                "port": "8080",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "httpd",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.httpd_deny": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "any:id.curl": ""
            }
          }
        ],
        "toPorts": [
          {
            "ports": [
              {
                "port": "9090",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "httpd_deny",
        "source": "unspec"
      }
    ]
  }
]
Revision: 19

17:38:30 STEP: Container "curl" test connectivity to "httpd1"
17:38:30 STEP: Container "curl" test connectivity to "httpd1"
17:38:30 STEP: Container "curl2" test connectivity to "httpd1"
17:38:35 STEP: Container "curl2" test connectivity to "httpd1"
17:38:40 STEP: Container "curl" test connectivity to "httpd2"
17:38:45 STEP: Container "curl" test connectivity to "httpd2"
17:38:50 STEP: Container "curl2" test connectivity to "httpd2"
17:38:55 STEP: Container "curl2" test connectivity to "httpd2"
17:39:01 STEP: Container "client" test connectivity to "server"
17:39:05 STEP: Container "client" test connectivity to "server"
17:39:09 STEP: Container "client" test connectivity to "server"
17:39:09 STEP: Container "client" test connectivity to "server"
17:39:09 STEP: Container "client" test connectivity to "server"
17:39:12 STEP: Container "client" test connectivity to "server"
17:39:15 STEP: Container "client" test connectivity to "server"
17:39:18 STEP: Container "client" test connectivity to "server"
17:39:22 STEP: Testing bidirectional connectivity from client to server
17:39:22 STEP: container server pinging client IPv6 (should NOT work)
17:39:31 STEP: container server pinging client IPv4 (should NOT work)
17:39:40 STEP: ============= Finished Connectivity Test ============= 
17:39:40 STEP: Testing with endpoint configuration option: ConntrackLocal=Enabled
17:39:47 STEP: ============= Starting Connectivity Test ============= 
17:39:47 STEP: Getting IPs of each spawned container
17:39:47 STEP: client container Docker networking: map["EndpointID":"dd78d04953ceaf3e6862e7e74d9e58dd7787207443bd02b79b6a340bf30f4eca" "IPv4":"10.15.3.86" "IPv6":"f00d::a0f:0:0:22e3" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"client" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:39:47 STEP: server container Docker networking: map["EndpointID":"7547a804e1172e1e103ad6cbeb9c55957e8dbad6e3350d2104b69ab970aae2b0" "IPv4":"10.15.147.41" "IPv6":"f00d::a0f:0:0:9b29" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"server" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:39:47 STEP: httpd1 container Docker networking: map["EndpointID":"54d2e0a87a5c1f91c254d1bb7022b506f0ed3d51277b6eb605b8576606478945" "IPv4":"10.15.63.56" "IPv6":"f00d::a0f:0:0:e2b4" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"httpd1" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:39:48 STEP: httpd2 container Docker networking: map["EndpointID":"bab6b0bad18a4e77baa156fe8dee345527e592b7e1d51f4c40c6029adc8af99f" "IPv4":"10.15.60.130" "IPv6":"f00d::a0f:0:0:f43a" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"httpd2" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:39:48 STEP: curl1 container Docker networking: map["EndpointID":"f8376f92c5ce585d315dcf061243f839f1743b5d0bd41d9e1b3e06c56d4f7401" "IPv4":"10.15.173.123" "IPv6":"f00d::a0f:0:0:c6c" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"curl" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:39:48 STEP: httpd1 container Docker networking: map["EndpointID":"b265856d5a542524ea13f60f093740274b7ff23b3123def85bbeea81601f8237" "IPv4":"10.15.158.213" "IPv6":"f00d::a0f:0:0:4d9a" "IPv6Gateway":"f00d::a0f:0:0:ee48" "Name":"curl2" "NetworkID":"cde0ed7b65cbd3b17577b6edd03d0a9a54d48a8d00a06dd57ef26884528063cc"]
17:39:48 STEP: Showing policies imported to Cilium
[
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.curl": ""
      }
    },
    "egress": [
      {
        "toPorts": [
          {
            "ports": [
              {
                "port": "80",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "curl",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.server": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "reserved:host": ""
            }
          },
          {
            "matchLabels": {
              "any:id.client": ""
            }
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "server",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.client": ""
      }
    },
    "egress": [
      {
        "toEndpoints": [
          {
            "matchLabels": {
              "any:id.server": ""
            }
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "egress-from-client-to-server",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.httpd": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "any:id.curl": ""
            }
          }
        ],
        "toPorts": [
          {
            "ports": [
              {
                "port": "80",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "httpd",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.httpd": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "any:id.curl2": ""
            }
          }
        ],
        "toPorts": [
          {
            "ports": [
              {
                "port": "8080",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "httpd",
        "source": "unspec"
      }
    ]
  },
  {
    "endpointSelector": {
      "matchLabels": {
        "any:id.httpd_deny": ""
      }
    },
    "ingress": [
      {
        "fromEndpoints": [
          {
            "matchLabels": {
              "any:id.curl": ""
            }
          }
        ],
        "toPorts": [
          {
            "ports": [
              {
                "port": "9090",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "labels": [
      {
        "key": "id",
        "value": "httpd_deny",
        "source": "unspec"
      }
    ]
  }
]
Revision: 19

17:39:48 STEP: Container "curl" test connectivity to "httpd1"
17:39:48 STEP: Container "curl" test connectivity to "httpd1"
17:39:48 STEP: Container "curl2" test connectivity to "httpd1"
17:39:53 STEP: Container "curl2" test connectivity to "httpd1"
17:39:58 STEP: Container "curl" test connectivity to "httpd2"
FAIL: The result of "curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 20 http://[f00d::a0f:0:0:f43a]:80 -w \"time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'\"" from container "curl" to httpd2 does not match
Expected
    <bool>: true
to be false
=== Test Finished at 2022-03-11T17:39:58Z====
17:39:58 STEP: Running JustAfterEach block for EntireTestsuite RuntimeConntrackInVethModeTest
FAIL: Found 1 Cilium logs matching list of errors that must be investigated:
panic:
===================== TEST FAILED =====================
17:40:00 STEP: Running AfterFailed block for EntireTestsuite RuntimeConntrackInVethModeTest
cmd: sudo cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                     
	 415        Disabled           Disabled          5085       container:id.httpd_deny       f00d::a0f:0:0:f43a   10.15.60.130    regenerating            
	                                                            container:id.service1                                                                      
	 592        Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:c02d   10.15.209.198   waiting-to-regenerate   
	 743        Disabled           Disabled          2243       container:id.client           f00d::a0f:0:0:22e3   10.15.3.86      restoring               
	 995        Disabled           Disabled          55488      container:id.httpd            f00d::a0f:0:0:e2b4   10.15.63.56     regenerating            
	                                                            container:id.service1                                                                      
	 1208       Disabled           Disabled          1          reserved:host                                                      ready                   
	 1416       Disabled           Disabled          26003      container:id.server           f00d::a0f:0:0:9b29   10.15.147.41    regenerating            
	 2042       Disabled           Disabled          24664      container:id.curl             f00d::a0f:0:0:c6c    10.15.173.123   regenerating            
	 2370       Disabled           Disabled          35214      container:id.curl2            f00d::a0f:0:0:4d9a   10.15.158.213   restoring               
	 
Stderr:
 	 

cmd: sudo cilium policy get
Exitcode: 0 
Stdout:
 	 []
	 Revision: 1
	 
Stderr:
 	 

cmd: sudo cilium endpoint list
Exitcode: 0 
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4            STATUS   
	            ENFORCEMENT        ENFORCEMENT                                                                                     
	 415        Disabled           Disabled          5085       container:id.httpd_deny       f00d::a0f:0:0:f43a   10.15.60.130    ready   
	                                                            container:id.service1                                                      
	 592        Disabled           Disabled          4          reserved:health               f00d::a0f:0:0:c02d   10.15.209.198   ready   
	 743        Disabled           Disabled          2243       container:id.client           f00d::a0f:0:0:22e3   10.15.3.86      ready   
	 995        Disabled           Disabled          55488      container:id.httpd            f00d::a0f:0:0:e2b4   10.15.63.56     ready   
	                                                            container:id.service1                                                      
	 1208       Disabled           Disabled          1          reserved:host                                                      ready   
	 1416       Disabled           Disabled          26003      container:id.server           f00d::a0f:0:0:9b29   10.15.147.41    ready   
	 2042       Disabled           Disabled          24664      container:id.curl             f00d::a0f:0:0:c6c    10.15.173.123   ready   
	 2370       Disabled           Disabled          35214      container:id.curl2            f00d::a0f:0:0:4d9a   10.15.158.213   ready   
	 
Stderr:
 	 

cmd: sudo cilium bpf policy get --all
Exitcode: 0 
Stdout:
 	 /sys/fs/bpf/tc/globals/cilium_policy_00415:
	 
	 POLICY   DIRECTION   LABELS (source:key[=value])   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
	 Allow    Ingress     reserved:unknown              ANY          NONE         608     6         
	 Allow    Egress      reserved:unknown              ANY          NONE         622     4         
	 
	 
	 /sys/fs/bpf/tc/globals/cilium_policy_00592:
	 
	 POLICY   DIRECTION   LABELS (source:key[=value])   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
	 Allow    Ingress     reserved:unknown              ANY          NONE         0       0         
	 Allow    Egress      reserved:unknown              ANY          NONE         0       0         
	 
	 
	 /sys/fs/bpf/tc/globals/cilium_policy_00743:
	 
	 POLICY   DIRECTION   LABELS (source:key[=value])   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
	 Allow    Ingress     reserved:unknown              ANY          NONE         0       0         
	 Allow    Egress      reserved:unknown              ANY          NONE         0       0         
	 
	 
	 /sys/fs/bpf/tc/globals/cilium_policy_00995:
	 
	 POLICY   DIRECTION   LABELS (source:key[=value])   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
	 Allow    Ingress     reserved:unknown              ANY          NONE         0       0         
	 Allow    Egress      reserved:unknown              ANY          NONE         0       0         
	 
	 
	 /sys/fs/bpf/tc/globals/cilium_policy_01208:
	 
	 POLICY   DIRECTION   LABELS (source:key[=value])   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
	 Allow    Ingress     reserved:unknown              ANY          NONE         0       0         
	 Allow    Egress      reserved:unknown              ANY          NONE         0       0         
	 
	 
	 /sys/fs/bpf/tc/globals/cilium_policy_01416:
	 
	 POLICY   DIRECTION   LABELS (source:key[=value])   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
	 Allow    Ingress     reserved:unknown              ANY          NONE         0       0         
	 Allow    Egress      reserved:unknown              ANY          NONE         0       0         
	 
	 
	 /sys/fs/bpf/tc/globals/cilium_policy_02042:
	 
	 POLICY   DIRECTION   LABELS (source:key[=value])   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
	 Allow    Ingress     reserved:unknown              ANY          NONE         622     4         
	 Allow    Egress      reserved:unknown              ANY          NONE         608     6         
	 
	 
	 /sys/fs/bpf/tc/globals/cilium_policy_02370:
	 
	 POLICY   DIRECTION   LABELS (source:key[=value])   PORT/PROTO   PROXY PORT   BYTES   PACKETS   
	 Allow    Ingress     reserved:unknown              ANY          NONE         0       0         
	 Allow    Egress      reserved:unknown              ANY          NONE         0       0         
	 
	 
	 
Stderr:
 	 goroutine 1 [running]:
	 goroutine 1 runtime/debug.Stack()
	 goroutine 1 	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
	 goroutine 1 github.com/cilium/cilium/pkg/lock.(*internalRWMutex).Unlock(0xc0006da3d8)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/lock/lock_debug.go:58 +0x92
	 goroutine 1 github.com/cilium/cilium/pkg/bpf.(*Map).Open(0xc0006da360)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bpf/map_linux.go:544 +0x95
	 goroutine 1 github.com/cilium/cilium/pkg/maps/policymap.Open({0xc00062b290, 0x4d5f25})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/pkg/maps/policymap/policymap.go:419 +0x2e
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.dumpMap({0xc00062b290, 0xc000134008})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_policy_get.go:85 +0x47
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.listAllMaps()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_policy_get.go:68 +0x185
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.glob..func23(0x3ba5ee0, {0xc0002aa930, 0x0, 0x1})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/bpf_policy_get.go:41 +0x4b
	 goroutine 1 github.com/spf13/cobra.(*Command).execute(0x3ba5ee0, {0xc0002aa920, 0x1, 0x1})
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
	 goroutine 1 github.com/spf13/cobra.(*Command).ExecuteC(0x3ba77e0)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
	 goroutine 1 github.com/spf13/cobra.(*Command).Execute(...)
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/vendor/github.com/spf13/cobra/command.go:902
	 goroutine 1 github.com/cilium/cilium/cilium/cmd.Execute()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/cmd/root.go:36 +0x25
	 goroutine 1 main.main()
	 goroutine 1 	/home/vagrant/go/src/github.com/cilium/cilium/cilium/main.go:16 +0x17
	 

===================== Exiting AfterFailed =====================
17:40:16 STEP: Running AfterEach for block EntireTestsuite RuntimeConntrackInVethModeTest
17:40:19 STEP: Setting PolicyEnforcement=default
17:40:19 STEP: Running AfterEach for block EntireTestsuite

[[ATTACHMENT|e804e2ff_RuntimeConntrackInVethModeTest_Conntrack-related_configuration_options_for_endpoints.zip]]
17:40:22 STEP: Running AfterAll block for EntireTestsuite RuntimeConntrackInVethModeTest

Resources

Anything else?

The CLI crash stack trace is likely not directly related to the failure, tracking that in #19119.

The failure here seems to be that the cilium-agent pod happened to fail BPF checks the first time it started up (below from cilium.log) which caused a delay in provisioning Cilium, so Cilium was not ready before proceeding with the test. While Cilium is regenerating policy for endpoints during startup, the curl failed.

Mar 11 17:33:59 runtime cilium-agent[24353]: level=fatal msg="BPF check: NOT OK." error="reading from bpftool feature probe stderr pipe failed: read |0: file already closed" subsys=linux-datapath

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/CIContinuous Integration testing issue or flakearea/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.ci/flakeThis is a known failure that occurs in the tree. Please investigate me!staleThe stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions