avoid unnecessary copy virtual services for sidecar scope calculation #41101
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After DeepCopy improvements, init context time takes roughly 39s, and more than 20% of cpu time is spent on the VirtualServicesForGateway function: https://github.com/istio/istio/blob/1.14.4/pilot/pkg/model/push_context.go#L863-L875 This function is called for every sidecar's egress host, for calculating the virtual services that are imported by the egress host. We have more than 10k sidecars and suppose each sidecar has 10 egress hosts, this function is called 100k times. What makes it worse is that all our virtual services are public (exportTo: *), so VirtualServicesForGateway creates and copies all virtual services (roughly also more than 10k) each time. This "make slice" and "slice copy" are expansive on such magnitude. This CL gets rid of such copy, instead of passing in the copied and merged version of the virtual services, just pass the virtualServiceIndex into the select function directly. This improves the init context time to roughly 25s. Change-Id: I48015e750a1019f12dfc35b0ca42b72fddfa87ba Reviewed-on: https://gerrit.musta.ch/c/public/istio/+/3745 Reviewed-by: Jungho Ahn <jungho.ahn@airbnb.com> Reviewed-by: Weibo He <weibo.he@airbnb.com>
|
cc @ramaraochavali I donot have bandwidth today, can you ptal |
airbnb-gerrit
pushed a commit
to airbnb/istio
that referenced
this pull request
Oct 4, 2022
This CL patches commit e40f57e from upstream istio into air-release-1.14.4 to improve propagation delay. Original PR: istio#41101 Change-Id: I7b0da42f591a6da9e20342235ccbf93f2741132b Reviewed-on: https://gerrit.musta.ch/c/public/istio/+/3822 Reviewed-by: Weibo He <weibo.he@airbnb.com>
airbnb-gerrit
pushed a commit
to airbnb/istio
that referenced
this pull request
Oct 12, 2022
Apply the following list of patches to istio 1.14.5: * sidecar: filter service ports to VS ports (istio#39067) * istio: register init push context metric (istio#40049) * istio: add metric for debouncing (istio#40523) * istio: fix PILOT_ENABLE_RDS_CACHE flag not working (istio#40719) * istio: support inline multi-values header in authz header match (https://gerrit.musta.ch/c/public/istio/+/3622, not yet merged upstream) * istio: improve deep copy for ServiceAttribute (istio#40966) * avoid unnecessary copy virtual services for sidecar scope calculation (istio#41101) Change-Id: Ia4c9bfd619a0eb38c1a829bff2efbd21fd3b9cb2 Reviewed-on: https://gerrit.musta.ch/c/public/istio/+/3883 Reviewed-by: Ying Zhu <ying.zhu@airbnb.com> Reviewed-by: Weibo He <weibo.he@airbnb.com>
airbnb-gerrit
pushed a commit
to airbnb/istio
that referenced
this pull request
Nov 10, 2022
Apply the following list of upstream commits to istio 1.15.3: * istio: add metric for debouncing (istio#40523) * istio: fix PILOT_ENABLE_RDS_CACHE flag not working (istio#40719) * istio: improve deep copy for ServiceAttribute (istio#40966) * avoid unnecessary copy virtual services for sidecar scope calculation (istio#41101) Change-Id: I2ee1d77d096a329dc8f590151223b37193dd4f1b Reviewed-on: https://gerrit.musta.ch/c/public/istio/+/3990 Reviewed-by: Ying Zhu <ying.zhu@airbnb.com> Reviewed-by: Ryan Smick <ryan.smick@airbnb.com>
|
/cherrypick release-1.15 |
|
@S-Chan: new pull request created: #42671 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
airbnb-gerrit
pushed a commit
to airbnb/istio
that referenced
this pull request
Feb 10, 2023
Apply the following list of upstream commits to istio 1.15.5: * istio: add metric for debouncing (istio#40523) * istio: improve deep copy for ServiceAttribute (istio#40966) * avoid unnecessary copy virtual services for sidecar scope calculation (istio#41101) Change-Id: I25f31e5633b77982606912bcb2ad2bc4e2da87f4 Reviewed-on: https://gerrit.musta.ch/c/public/istio/+/4381 Reviewed-by: Weibo He <weibo.he@airbnb.com> Reviewed-by: Stephen Chan <stephen.chan@airbnb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please provide a description of this PR:
Our production init push context time is very long (roughly 40s), and CPU profile shows that more than 20% of cpu time is spent on the VirtualServicesForGateway function:
https://github.com/istio/istio/blob/1.14.4/pilot/pkg/model/push_context.go#L863-L875
This function is called for every sidecar's egress host, for calculating the virtual services that are imported by the egress host. We have more than 10k sidecars and suppose each sidecar has 10 egress hosts, this function is called 100k times. This "make slice" and "slice copy" are expansive on such magnitude.
Such copy is unnecessary though. Instead of passing in the copied and merged version of the virtual services, just pass the virtualServiceIndex into the select function directly. From our testing, this improves the init context time to roughly 25s.
To help us figure out who should review this PR, please put an X in all the areas that this PR affects.
Please check any characteristics that apply to this pull request.