Skip to content

Contour occasionally gets stuck in a non-ready state and fails to start the XDS server #6613

@tsaarni

Description

@tsaarni

Contour occasionally gets stuck in a non-ready state and fails to start the XDS server. This issue is uncommon and occurs only under the following conditions:

  • --watch-namespaces flag is set to monitor a single namespace.
  • That namespace has only "unrelated" secrets that Contour does not process.
  • The stuck Contour instance is the follower (not leader).

To reproduce this issue, follow these steps:

  1. Deploy contour:
$ kubectl apply -f https://projectcontour.io/quickstart/contour.yaml
  1. Create a nearly empty namespace with one generic secret:
$ kubectl create ns almost-empty
$ kubectl -n almost-empty create secret generic my-secret --from-literal=secret=value
  1. Configure Contour to watch only the almost-empty namespace:
$ kubectl patch deployment contour -n projectcontour --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--watch-namespaces=almost-empty"}]'
  1. Restart both Contour instances by scaling down and back up:
$ kubectl -n projectcontour scale deployment --replicas=0 contour
$ kubectl -n projectcontour scale deployment --replicas=2 contour
  1. Verify that one Contour instance remains stuck in a non-ready state:
$ kubectl -n projectcontour get pod -l app=contour
NAME                      READY   STATUS    RESTARTS   AGE
contour-db59b775d-9k4wl   1/1     Running   0          4m17s
contour-db59b775d-tnqxm   0/1     Running   0          4m17s

The problem is a race condition, and its likelihood decreases if more resources (or no resources) are created in the watched namespace.

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions