-
Notifications
You must be signed in to change notification settings - Fork 8.1k
Add extra addresses for listeners configuration to support dual stack in Istio #40634
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
cc @adams-shaun can you help us review this? Do you have any ideas for how we can add tests to this? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Conceptually looks good. I think the way the code is structured right now makes it very error prone for use, I think we need to work on cleaning it up a bit.
|
||
// Build the "virtual" inbound listener. This will capture all inbound redirected traffic and contains: | ||
// * Passthrough filter chains, matching all unmatched traffic. There are a few of these to handle all cases | ||
// * Service filter chains. These will either be for each Port exposed by a Service OR Sidecar.Ingress configuration. | ||
allChains := buildInboundPassthroughChains(lb) | ||
allChains = append(allChains, chains...) | ||
return lb.buildInboundListener(model.VirtualInboundListenerName, util.BuildAddress(actualWildcard, ProxyInboundListenPort), false, allChains) | ||
l := lb.buildInboundListener(model.VirtualInboundListenerName, util.BuildAddress(actualWildcard, ProxyInboundListenPort), false, allChains) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we do this in buildInboundListener?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @howardjohn, I think it's better to separate the default service and extra service address, because 1. no change for buildInboundListener
; 2. no impact for single stack logic; 3. Extra service address should be added separately, because it's not just for dual stack, it may be used for other in future. Any idea?
extraAddress := &listener.AdditionalAddress{ | ||
Address: util.BuildAddress(exbd, uint32(opts.port.Port)), | ||
} | ||
res.AdditionalAddresses = append(res.AdditionalAddresses, extraAddress) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need a check for Envoy version, this field is new? Or its already in 1.15 and this PR will only be in 1.16?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @howardjohn, yes, AdditionalAddresses
is a new field. And yes, but how about to check the Istio version?
BTW, when I execute cmd: make build
it's okay with the new field of AdditionalAddresses
, but when I execute cmd make test
, there are errors message like below:
{Failed listener_builder_test.go:79: listener 127.0.0.1_15007 is invalid: invalid Listener.AdditionalAddresses[0]: embedded message failed validation | caused by: invalid AdditionalAddress.Address: embedded message failed validation | caused by: invalid Address.SocketAddress: embedded message failed validation | caused by: invalid SocketAddress.Address: value length must be at least 1 runes}
So I think make build
or make test
have different dependency libs version for github.com/envoyproxy/go-control-plane/envoy/config/listener/v3
Any idea?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a new function named IsIstioVersionGE116
to make sure that the field of AdditionalAddresses
can be added if istioctl version is greater or equal 1.16.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need go.mod updated to use envoyproxy/go-control-plane@2cb7986
Try running `go get github.com/envoyproxy/go@2cb7986b23ccfde4a77c26ebf3bfbfa71126fafd for your PR.
I have a PR to update the to a version of go.mod, but it's failing due to other reasons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It appears that updating the api is also blocked on envoyproxy/envoy#22804 as definitions were removed that Istio relies on for other features. Maybe you can get Alex to help you Steve to get it pushed through Envoy. This is blocking Istio as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @jacob-delgado, thanks, this error has been fixed!
// if it's dual stack environment | ||
if bindAddress != "" && exBindAddresses != "" { | ||
cc.bind = bindAddress | ||
cc.exBind = []string{exBindAddresses} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIRC before in dual stack we made 2 listeners. Now are we creating 2 listeners with 2 addresses each? Or did e remove the 2 listeners somewhere I missed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @howardjohn, according to original solution, there are 2 listeners for a service with both IPv4 and IPv6 addresses. For current solution, there is only 1 listener and there are address and additional addresses (new field) for it. The former is for the first address of the service, and the latter is for the rest addresses of the service.
Hi @howardjohn, @hzxuzhonghu and @ramaraochavali , I added the unit test and fixed the issues. And here is the new configdump file |
bind := egressListener.IstioListener.Bind | ||
var bind string | ||
var extraBind []string | ||
bind = egressListener.IstioListener.Bind |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bind = egressListener.IstioListener.Bind | |
bind := egressListener.IstioListener.Bind |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay, done!
// and make sure that it's necessary to add the extra addresses, such as it's a dual stack env | ||
if actualLocalHostAddrIPv6 != "" { | ||
if bind == actualLocalHostAddrIPv4 { | ||
listenerOpts.extraBind = []string{actualLocalHostAddrIPv6} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we want to make bind
and extraBind
? Should we just have bind
be a list and we automatically make them "extra"?
In envoy, is there any difference between the primary address and the "extra" addresses?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we want to make
bind
andextraBind
? Should we just havebind
be a list and we automatically make them "extra"?In envoy, is there any difference between the primary address and the "extra" addresses?
Hi @howardjohn, as discussing in previous PR, ClusterVIPs
can holds all IP instances which is available both for bind
and extraBind
.
Yes, according to the issue envoyproxy/envoy#11184 in envoy, additinal_addresses
can hold all other IP instances for one specified listener, except the primary one. And here is the API definition for it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@howardjohn For example, if there are 2 IP instances ["127.0.0.1", "::1"], then the primary address should be 127.0.0.1
, and the extra addresses should be [::1
]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get that's how it's implemented but my question is why? Do we really need to carry around a "address" and "extra addresses" or just a list of addresses and stick a random one as the main on in envoy config?
My concern with this PR, as noted before, is not the generates XDS but the large complexity we are adding to the code for something relatively simple, and lack of integration tests.
If we need all of these complex if statements and juggling around 4 address (localhost + wildcard x v4 v6) it's complex and error prone. Maybe this logic can be factored out into a struct, etc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @howardjohn, how about to factor the extra addresses adding logic into the function of structs buildListenerOpts
and inboundChainConfig
, then the similar if statements, such as if len(cc.extraBind) > 0 && util.IsIstioVersionGE116(lb.node.IstioVersion)
will be eliminated.
For integration test, the dual stack integration test will be added by @jacob-delgado
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get that's how it's implemented but my question is why? Do we really need to carry around a "address" and "extra addresses" or just a list of addresses and stick a random one as the main on in envoy config?
Hi @howardjohn, for this part, current solution has the similar implementation with the API definition of envoy: one primary address and some extra addresses. So it should be more reasonable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need to make all of our code much more complex just because envoy has a weird API, imo. We should pick the right API for our usage and then translate it to envoy as needed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Integration tests are an expectation for all features in Istio; please have them alongside the PR instead of afterwards. We can wait. If you want to merge without integration tests it can be done in the experimental branch
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't need to make all of our code much more complex just because envoy has a weird API, imo. We should pick the right API for our usage and then translate it to envoy as needed
Sure, agree with this idea, I will make the code as simple as possible, then re-submit it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Integration tests are an expectation for all features in Istio; please have them alongside the PR instead of afterwards. We can wait. If you want to merge without integration tests it can be done in the experimental branch
@howardjohn Sure, I will add integration tests for this feature, and make sure the additional_addresses
can be correctly added. Thanks for the comment here.
Hi @howardjohn, could you please help to approve this PR if it's okay? Thanks! ^_^ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have integration test which has dual nics?
Hi @hzxuzhonghu , that's a good question, in fact @jacob-delgado is focusing on building the integration test environment for dual stack. |
The
Why are we seeing such a huge increase in config size? |
Hi @howardjohn, thanks for your comparison above, then I see now. and I also checked the comparison between According to current analysis, this PR does not cause the increasing in configuration size if it drops the bech_test's |
/test benchmark |
wcLh.localHostIPv4 = LocalhostAddress | ||
wcLh.localHostIPv6 = LocalhostIPv6Address | ||
default: | ||
panic("Unknow IP mode") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
panic("Unknow IP mode") | |
panic("Unknown IP mode") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
bind := egressListener.IstioListener.Bind | ||
if bind == "" { | ||
if bindToPort { | ||
bind = actualLocalHostAddress | ||
bind = actualLocalHosts[0] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need extraBind
at all? Instead of bind
being a list?
I am very worried that somewhere else we will assume there is one bind and mutate it.
For example:
bind := 127.0.0.1
extraBind := ::1
... lot of code later ..
bind = SomeOtherIp // Oops! forgot to change extrabind!
bindToPort: bindToPort, | ||
} | ||
if len(extraBind) > 0 { | ||
listenerOpts.extraBind = append(listenerOpts.extraBind, extraBind...) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bindToPort: bindToPort, | |
} | |
if len(extraBind) > 0 { | |
listenerOpts.extraBind = append(listenerOpts.extraBind, extraBind...) | |
} | |
bindToPort: bindToPort, | |
extraBind: extraBind, | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
pilot/pkg/networking/util/util.go
Outdated
} | ||
return | ||
} | ||
log.Warnf("warning: No extra addresses need to be added to listener [%s]", ll.Name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally LGTM. Couple of nits. Still not clear what is the issue with bench marks though
pilot/pkg/model/context.go
Outdated
// wildCardIPs contains the wildcard host IPs, including IPv4 and IPv6 | ||
wildCardIPs []string | ||
|
||
// localHostIPs contains the wildcard host IPs, including IPv4 and IPv6 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: localHostIPs contains the local host IPs,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
bind := egressListener.IstioListener.Bind | ||
if bind == "" { | ||
if bindToPort { | ||
bind = actualLocalHostAddress | ||
bind = actualLocalHosts[0] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you please add a comment on why we are using bind = actualLocalHosts[0]?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One question, when user specify Bind explicitly, will we add additional Bind now?
pilot/pkg/model/context.go
Outdated
} | ||
node.GlobalUnicastIP = networkutil.GlobalUnicastIP(node.IPAddresses) | ||
node.wildCardIPs = wildCardIPs | ||
node.localHostIPs = localHostIPs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I donot feel very comfortable caching same or limited number of strings per proxy. From memory view, this is not friendly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @hzxuzhonghu, in fact they are not caching same, the content for wildCardIPs
and localHostIPs
depends on the service deployment, it can be, for example to localHostIPs
, ["127.0.0.1"], ["::1"] or ["127.0.0.1", "::1"].
BTW,
- if not store them in proxy, then Istio must calculate them,
- if not calculate them, then must store them
One of them should be choose.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @hzxuzhonghu, use 2 map to maintain wildCardIPs
and localHostIPs
now
pilot/pkg/model/context.go
Outdated
// Wildcards returns the wildCard IP addresses for current proxy | ||
func (node *Proxy) Wildcards() []string { | ||
if len(node.wildCardIPs) == 0 { | ||
node.DiscoverIPMode() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no call it every, this can be called in init proxy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @hzxuzhonghu, yeah, DiscoverIPMode
would be called one time if len(node.wildCardIPs) == 0
or len(node.localHosts) == 0
.
In fact, I'm afraid that DiscoverIPMode
is not called, so just in case. Do you suggest that remove the calling in both Wildcards()
and Localhosts()
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed!
bind = actualWildcards[0] | ||
if len(actualWildcards) > 1 { | ||
extraBind = actualWildcards[1:] | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we cache the result of bind and extraBind or quickly inferred somewhere
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @hzxuzhonghu, I think this may be the same as above answer of mine.
@@ -389,12 +401,13 @@ func (lb *ListenerBuilder) buildSidecarOutboundListeners(node *model.Proxy, | |||
bind: bind, | |||
port: listenPort, | |||
bindToPort: bindToPort, | |||
extraBind: extraBind, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about additionalBind
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the extraBind
will be translated into additional addresses for listener in later.
pilot/pkg/networking/util/util.go
Outdated
@@ -173,6 +173,22 @@ func BuildAddress(bind string, port uint32) *core.Address { | |||
} | |||
} | |||
|
|||
// BuildExtraAddresses can add extra addresses to additional addresses for a listener | |||
func BuildExtraAddresses(extrAddresses []string, listenPort uint32, ll *listener.Listener, node *model.Proxy) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggest moving listener param here, this function should be simple
listener.AdditionalAddresses = BuildExtraAddresses()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay, I understand, generate the additional addresses in BuildExtraAddresses
and remove the listener parameter, then specify as listener.AdditionalAddresses = BuildExtraAddresses()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
Hi @hzxuzhonghu, what do you mean |
Not sure this is right in this case |
Hi @hzxuzhonghu, I re-submitted, PTAL, thanks for your comments! ^_^ |
/test integ-security-multicluster |
Hi @hzxuzhonghu, re-submitted, maintain 2 maps to fetch host addresses according to proxy’s IP mode. Thanks for your comments! ^_^ |
pilot/pkg/networking/util/util.go
Outdated
@@ -173,6 +173,23 @@ func BuildAddress(bind string, port uint32) *core.Address { | |||
} | |||
} | |||
|
|||
// BuildExtraAddresses can add extra addresses to additional addresses for a listener | |||
func BuildExtraAddresses(extrAddresses []string, listenPort uint32, node *model.Proxy) []*listener.AdditionalAddress { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
replace Extra with Additional
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay, no problem!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
Please provide a description of this PR:
Add extra addresses for listeners configuration to support dual stack in Istio, this PR is part of issue#40394
This PR just focus on normal k8s service in service mesh, and add the additional addresses for specified listener of service. There fore, it does not include below:
Below snapshots are test results and the attach contains the whole configdump file exclude eds via
curl localhost:15000/config_dump
Inbound listener additional addresses:

Outbound listener additional addresses:

Application/Service listener additional addresses:

Config dump:
extra_addresses_configdump.log
Note: I did not add any test case and I will add necessary test case after all the business code changes are available for reviewers.