cilium, gops: remap to fixed port to avoid collision with nodeport range #14329
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
test-me-please |
rolinh
requested changes
Dec 9, 2020
Ah, good point, that one fell through the cracks :/ will add. |
aanm
requested changes
Dec 9, 2020
There was a problem hiding this comment.
the overall changes LGTM, however a couple points that we need to take care:
- It would likely be better to
Fatalif we can't initialize gops. If gops isn't running at all we won't be able to debug Cilium if we need it; - Currently, the default is
127.0.0.1:0and we are changing it tolocalhost:XXXX, maybe we should keep the127.0.0.1. - I don't see a "reuse port" in gops, so it is likely that we will hit something similar as #11573.
|
Marked for backport to v1.8 as this also fixes an issue with port collision between gops agent and the proxy, see #13400. I'll send a manual backport PR as cherry-picking is not straight forward. |
|
1.8 backport PR: #15634 |
|
Apologies, I forgot to mark this as |
aanm
pushed a commit
to tklauser/cilium
that referenced
this pull request
Apr 17, 2021
[ upstream commit 7757d31 ] Manually backported from cilium#14329 to address cilium#13400 for v1.8. Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: cilium#14218 Reported-by: Lee Hu via Slack Co-authored-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Tobias Klauser <tobias@cilium.io>
aanm
pushed a commit
that referenced
this pull request
Apr 17, 2021
[ upstream commit 7757d31 ] Manually backported from #14329 to address #13400 for v1.8. Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: #14218 Reported-by: Lee Hu via Slack Co-authored-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Tobias Klauser <tobias@cilium.io>
EricMountain
pushed a commit
to DataDog/cilium
that referenced
this pull request
Feb 21, 2022
[ upstream commit 7757d31 ] Manually backported from cilium#14329 to address cilium#13400 for v1.8. Lee reported that kube-proxy log had a warning that its bind protection couldn't bind a specific port in the nodeport range. Turns out gops was using this particular port already through it's auto-binding (127.0.0.1:0). Meaning that in case gops collides with a NodePort service, we might not be able to pull gops data from that port since either kube-proxy or kube-proxt free variant will redirect us to the actual service instead. Given this is rather unpredictable wrt which port the agent will bind for gops, remap it to a fixed default port and add a user configurable knob that allows to use a different one if necessary. Given the agent, operator, clustermesh-apiserver and hubble-relay all start the gops listener, add the --gops-port flag to each of them. The CNI does not have gops enabled by default but only in debug mode hence no changes there for now given it's unlikely being used this way in production. Fixes: cilium#14218 Reported-by: Lee Hu via Slack Co-authored-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Tobias Klauser <tobias@cilium.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See commit msg.