-
Notifications
You must be signed in to change notification settings - Fork 8k
Description
Bug Description
Issue 1
When istio proxy comes up for the first time, and it is given certificates via SDS, it was observed that it stores duplicate entries of the certificate as seen in the /certs
endpoint.
Issue 2
Envoy continues to listen on stale certificates, even when /certs
endpoint shows that it has new certificates. One thing to note here is, that the /certs
endpoint shows both new as well as old certificates.
Envoy is serving stale certificates only when there are more than 1 unique certificates found in the /certs
endpoint.
Version
$ istioctl version
1.13.4
$ kubectl version
1.21 (both)
Additional data points
It is consistently seen that the /certs
endpoints shows constant occurrences of the old certificate:
k exec -it debug-62rg9 -c istio-proxy -- `echo $certs_json` | grep cert_chain -A 11 | grep c34eeb5c60ab46b7b7afd046aab7823c | wc -l
39
Even when certificates rotate, the count of the original certificate remains consistent in /certs
endpoint.
Some parsed data from the /certs
endpoint:
╰─$ cat ratings-v1-b6994bb9-vjhtq.certs.json| grep cert_chain | wc -l
29
╭─aaeron@macos-C02G72ZBMD6T ~/workspace/setup/istio/releases/istio-1.13.4
╰─$ cat ratings-v1-b6994bb9-vjhtq.certs.json| grep cert_chain -A 3 | grep serial | wc -l
29
╭─aaeron@macos-C02G72ZBMD6T ~/workspace/setup/istio/releases/istio-1.13.4
╰─$ cat ratings-v1-b6994bb9-vjhtq.certs.json| grep cert_chain -A 3 | grep serial | sort -u
"serial_number": "26931ad6c653ea029212660e6edb781c",
The above command show that there are 29 instances of cert_chain, even though the serial number of all the certificates are the same. Which means that /certs
endpoint is storing duplicate entries.
Data from /certs
endpoint in cluster created via kind.
{
"certificates": [
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
},
{
"ca_cert": [
{
"path": "\u003cinline\u003e",
"serial_number": "e2110e139cc6cc7a",
"subject_alt_names": [],
"days_until_expiration": "34863",
"valid_from": "2018-01-24T19:15:51Z",
"expiration_time": "2117-12-31T19:15:51Z"
}
],
"cert_chain": [
{
"path": "\u003cinline\u003e",
"serial_number": "26931ad6c653ea029212660e6edb781c",
"subject_alt_names": [
{
"uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
}
],
"days_until_expiration": "0",
"valid_from": "2022-07-19T05:18:54Z",
"expiration_time": "2022-07-20T05:20:54Z"
}
]
}
]
}