Proxy: Do not release acknowledged proxy ports when endpoint regeneration fails #36142
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
|
/test |
1 task
nebril
approved these changes
Nov 25, 2024
|
/test |
3658d6e to
2f4bc6d
Compare
|
/test |
sayboras
approved these changes
Nov 26, 2024
|
/test |
93dddbb to
b54f3f8
Compare
|
/test |
tommyp1ckles
approved these changes
Dec 4, 2024
jrajahalme
added a commit
to jrajahalme/cilium
that referenced
this pull request
Dec 7, 2024
Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgement is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgement status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: cilium#36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
jrajahalme
added a commit
to jrajahalme/cilium
that referenced
this pull request
Dec 7, 2024
Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgment is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgment status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: cilium#36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
jrajahalme
added a commit
to jrajahalme/cilium
that referenced
this pull request
Dec 7, 2024
Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgment is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgment status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: cilium#36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
jrajahalme
added a commit
to jrajahalme/cilium
that referenced
this pull request
Dec 7, 2024
Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgment is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgment status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: cilium#36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 9, 2024
Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgment is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgment status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: #36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
jrajahalme
added a commit
to jrajahalme/cilium
that referenced
this pull request
Dec 9, 2024
[ upstream commit d79ab31 ] Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgment is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgment status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: cilium#36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
5 tasks
jrajahalme
added a commit
to jrajahalme/cilium
that referenced
this pull request
Dec 9, 2024
[ upstream commit d79ab31 ] Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgment is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgment status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: cilium#36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
jrajahalme
added a commit
to jrajahalme/cilium
that referenced
this pull request
Dec 9, 2024
[ upstream commit d79ab31 ] Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgment is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgment status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: cilium#36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 10, 2024
[ upstream commit d79ab31 ] Take a proxy port reference for new redirects as soon as they are created, rather than when an acknowledgment is received. This fixes the case where a delayed release of the proxy port takes effect after a proxy redirect has been created but not acknowledged yet, signified by these error logs seen in the CI: msg="Created new proxy instance" ProxyPort=19659 id="2424:egress:TCP:64:" l7parser=http ... msg="Delayed release of proxy port 19659" id=cilium-http-egress ... msg="Datapath proxy redirection cannot be enabled, L7 proxy may be bypassed" error="ackProxyPort: zero port on cilium-http-egress not allowed" id="2424:egress:TCP:4096:" l7parser=http ... This change makes it necessary to manage the acknowledgment status of a proxy port separately from the 'nRedirects' field. Add a new 'acknowledged' field for this purpose. Fixes: #36142 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
joestringer
reviewed
Dec 10, 2024
Comment on lines
+130
to
+133
| os.Mkdir(bootstrapDir, 0777) | ||
|
|
||
| // Make sure sockets dir exists | ||
| os.Mkdir(GetSocketDir(config.runDir), 0777) |
There was a problem hiding this comment.
@jrajahalme should this use defaults.RuntimePathRights instead (0775, ie avoid write access for every user on the node, given that this code runs in production environments)?
I briefly looked at proposing the change myself, but the commit seems to suggest that some magic is necessary in order to run the corresponding tests. Not sure how I would validate that this doesn't immediately break the test again.
This was referenced Dec 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Endpoint regeneration can fail for many reasons, only one of which is Envoy NACKing the listener configuration. Previously it was possible for a proxy port (e.g., DNS proxy port) to be released (and subsequently reallocated) if endpoint regeneration adding a new redirect (such as endpoint restore on agent restart) happened to fail for any reason. Handle proxy port ACK/NACK directly with an AddListener callback and only reset the proxy port if there are no redirects on the port and a NACK was received. Since envoy.AddListener is not used for DNS proxy redirects, it follows that DNS proxy port is now never reset. The AddListener callback allows to keep even newly configured redirects if the regeneration failure reason is not a NACK from Envoy. This helps reduce churn on Envoy listener configuration and the associated datapath proxy port redirection when endpoint regeneration fails for other reasons than Envoy Listener NACK.
To make the AddListener callback work we had to add a separate mutex for the proxy ports, so that a proxy mutex can be held without holding the proxy ports mutex while calling AddListener with a callback that then needs to take the proxy ports mutex. This also happens to solve a latent deadlock bug in CEC parser that also needs to take the mutex for acknowledging a proxy port. In this case we had different locking order for the
Proxy.mutexandxdsServer.mutexin two different code paths:upsertListener()xdsServer.mutexProxy.mutexin the callback viaAckProxyPort()Proxy.mutexto create a new redirectAddListener()which locksxdsServer.mutexWith the new
Proxy.proxyPorts.mutexthe above is modified so thatAckProxyPort()no longer needs to takeProxy.mutex(butProxy.proxyPorts.mutexinstead), and on the latter pathAddListener()is called without holding the newProxy.proxyPorts.mutexwhich can then be taken in the callback function as needed.Fixes: #36063