Skip to content

Recompute service account at endpoint shard when all endpoints are removed #36882

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 21, 2022

Conversation

bianpengyuan
Copy link
Contributor

@bianpengyuan bianpengyuan commented Jan 18, 2022

fix #36465 and #31534.

The sequence to trigger the issue:

  1. Scale down a service's workloads to zero, either manually, or some upgrade or reschedule event. This will trigger an EDS push, which does not update Cluster / push CDS.
  2. Update any istio config or any service, which will trigger a Cluster update and CDS push for all services (Note CDS cache is invalidated at step 1, which means Cluster for this service will be recomputed), and for this service, since there is no endpoint, the default validation context / SAN match will be empty "default_validation_context": {}.
  3. Now scale up the workloads, this will trigger another EDS push, and we have the logic to check if the service accounts have any update (
    saUpdated := s.UpdateServiceAccount(ep, hostname)
    ). if there is, a CDS update will also be triggered. However, whether the service account change is checked against the EndpointShards cache, which is not cleaned up when all endpoints get removed at step 1, so no CDS update will happen.

@bianpengyuan bianpengyuan requested a review from a team as a code owner January 18, 2022 21:41
@istio-testing istio-testing added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jan 18, 2022
@bianpengyuan
Copy link
Contributor Author

/retest

@bianpengyuan bianpengyuan changed the title Recompute service account at endpoint shard when all endpoints are re… Recompute service account at endpoint shard when all endpoints are removed Jan 18, 2022
Copy link
Member

@howardjohn howardjohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, LGTM but will let @ramaraochavali or @hzxuzhonghu take a look since I think they are familiar with this code deeply.

@bianpengyuan
Copy link
Contributor Author

/retest

@bianpengyuan
Copy link
Contributor Author

@howardjohn ping for release notes approval. thanks!

@istio-testing
Copy link
Collaborator

In response to a cherrypick label: new pull request created: #36939

@istio-testing
Copy link
Collaborator

In response to a cherrypick label: new pull request created: #36940

howardjohn added a commit to howardjohn/istio that referenced this pull request Sep 8, 2022
Fixes istio#39652

This reverts istio#36882. At the time,
that PR was needed because EDS ServiceAccounts and CDS ServiceAccounts
were decoupled; Since istio#39133, this is
no longer true, and the fix in 36882 is not needed any longer.

This PR *removes* the test added in 36882 (since it tests low level
details that are not relevant anymore). It improves the existing
TestEndpointFlipFlops test -- while that test *would* have caught the
regression, it didn't actually set any service accounts so it was
missed. The update changes it to correctly detect the behavior (it now
fails without this PR, passes with it).
istio-testing pushed a commit that referenced this pull request Sep 12, 2022
* Disable full push on scale from 1->0->1

Fixes #39652

This reverts #36882. At the time,
that PR was needed because EDS ServiceAccounts and CDS ServiceAccounts
were decoupled; Since #39133, this is
no longer true, and the fix in 36882 is not needed any longer.

This PR *removes* the test added in 36882 (since it tests low level
details that are not relevant anymore). It improves the existing
TestEndpointFlipFlops test -- while that test *would* have caught the
regression, it didn't actually set any service accounts so it was
missed. The update changes it to correctly detect the behavior (it now
fails without this PR, passes with it).

* add note

* fix note

* Add new SA case
howardjohn added a commit to howardjohn/istio that referenced this pull request Sep 12, 2022
* Disable full push on scale from 1->0->1

Fixes istio#39652

This reverts istio#36882. At the time,
that PR was needed because EDS ServiceAccounts and CDS ServiceAccounts
were decoupled; Since istio#39133, this is
no longer true, and the fix in 36882 is not needed any longer.

This PR *removes* the test added in 36882 (since it tests low level
details that are not relevant anymore). It improves the existing
TestEndpointFlipFlops test -- while that test *would* have caught the
regression, it didn't actually set any service accounts so it was
missed. The update changes it to correctly detect the behavior (it now
fails without this PR, passes with it).

* add note

* fix note

* Add new SA case

(cherry picked from commit 9f1cbfb)
istio-testing pushed a commit that referenced this pull request Sep 13, 2022
* SAN improvements (#40863)

* wip

* wip

* Add tests

* drop logs

* revert changes

* Add one more case

* Add namespace into SA key as well

* Add back svc.ServiceAccounts

* add note

* lint

(cherry picked from commit 0abe39d)

* Disable full push on scale from 1->0->1 (#40866)

* Disable full push on scale from 1->0->1

Fixes #39652

This reverts #36882. At the time,
that PR was needed because EDS ServiceAccounts and CDS ServiceAccounts
were decoupled; Since #39133, this is
no longer true, and the fix in 36882 is not needed any longer.

This PR *removes* the test added in 36882 (since it tests low level
details that are not relevant anymore). It improves the existing
TestEndpointFlipFlops test -- while that test *would* have caught the
regression, it didn't actually set any service accounts so it was
missed. The update changes it to correctly detect the behavior (it now
fails without this PR, passes with it).

* add note

* fix note

* Add new SA case

(cherry picked from commit 9f1cbfb)
yxun pushed a commit to yxun/maistra-istio that referenced this pull request Sep 28, 2023
* Disable full push on scale from 1->0->1

Fixes istio/istio#39652

This reverts istio/istio#36882. At the time,
that PR was needed because EDS ServiceAccounts and CDS ServiceAccounts
were decoupled; Since istio/istio#39133, this is
no longer true, and the fix in 36882 is not needed any longer.

This PR *removes* the test added in 36882 (since it tests low level
details that are not relevant anymore). It improves the existing
TestEndpointFlipFlops test -- while that test *would* have caught the
regression, it didn't actually set any service accounts so it was
missed. The update changes it to correctly detect the behavior (it now
fails without this PR, passes with it).

* add note

* fix note

* Add new SA case
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
4 participants