Bootstrap flag for connection limit overload manager resource monitor #6794

seth-epps · 2024-12-06T19:27:48Z

Overview

Manual testing

Setting the configuration parameters for the enforced connection limit listener and the connection limits

diff --git a/examples/contour/01-contour-config.yaml b/examples/contour/01-contour-config.yaml
index 6eb7720b..d56b811c 100644
--- a/examples/contour/01-contour-config.yaml
+++ b/examples/contour/01-contour-config.yaml
@@ -184,3 +184,7 @@ data:
     #  socket-options:
     #    tos: 64
     #    traffic-class: 64
+    #
+    omEnforcedHealthListener:
+      address: 0.0.0.0
+      port: 8003
diff --git a/examples/contour/03-envoy.yaml b/examples/contour/03-envoy.yaml
index b9c71f39..6e4e474d 100644
--- a/examples/contour/03-envoy.yaml
+++ b/examples/contour/03-envoy.yaml
@@ -105,6 +105,7 @@ spec:
         - --envoy-cafile=/certs/ca.crt
         - --envoy-cert-file=/certs/tls.crt
         - --envoy-key-file=/certs/tls.key
+        - --overload-dowstream-max-conn=100 # arbitrary value
         command:
         - contour
         image: ghcr.io/projectcontour/contour:main

After running make install-contour-working

# kubectl -n projectcontour port-forward envoy-hq4bj 9001:9001
#
# check listener configs
#
# default stats listner (should ignore -- this is for liveness)
❯ curl -s localhost:9001/config_dump | jq '.configs[] |
  select(."@type" == "type.googleapis.com/envoy.admin.v3.ListenersConfigDump") |
  .dynamic_listeners[] | select(.name == "stats-health").active_state.listener |
  "port: \(.address.socket_address.port_value), ignore_conn_limit: \(.ignore_global_conn_limit)"'
"port: 8002, ignore_conn_limit: true"
#
# enforced connection limit listener (should enforce [null] -- this is for readiness)
#
❯ curl -s localhost:9001/config_dump | jq '.configs[] |
  select(."@type" == "type.googleapis.com/envoy.admin.v3.ListenersConfigDump") |
  .dynamic_listeners[] | select(.name == "health-om-enforced").active_state.listener |
  "port: \(.address.socket_address.port_value), ignore_conn_limit: \(.ignore_global_conn_limit)"'
"port: 8003, ignore_conn_limit: null"
#
# check bootstrap OM config
#
❯ curl -s localhost:9001/config_dump | jq '.configs[] | select(."@type" == "type.googleapis.com/envoy.admin.v3.BootstrapConfigDump") | .bootstrap.overload_manager'
{
  "refresh_interval": "0.250s",
  "resource_monitors": [
    {
      "name": "envoy.resource_monitors.global_downstream_max_connections",
      "typed_config": {
        "@type": "type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig",
        "max_active_downstream_connections": "100"
      }
    }
  ]
}

codecov · 2024-12-06T20:01:15Z

Codecov Report

Attention: Patch coverage is 96.72131% with 4 lines in your changes missing coverage. Please review.

Project coverage is 80.77%. Comparing base (f140c71) to head (45f46d4).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/contour/contour.go	0.00%	3 Missing ⚠️
cmd/contour/serve.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6794      +/-   ##
==========================================
+ Coverage   80.74%   80.77%   +0.03%     
==========================================
  Files         131      131              
  Lines       19884    19936      +52     
==========================================
+ Hits        16055    16104      +49     
- Misses       3537     3540       +3     
  Partials      292      292

Files with missing lines	Coverage Δ
cmd/contour/bootstrap.go	`100.00% <100.00%> (ø)`
cmd/contour/servecontext.go	`84.94% <100.00%> (+0.03%)`	⬆️
internal/envoy/bootstrap.go	`57.14% <100.00%> (+4.87%)`	⬆️
internal/envoy/v3/bootstrap.go	`93.00% <100.00%> (+0.38%)`	⬆️
internal/envoy/v3/stats.go	`100.00% <100.00%> (ø)`
internal/featuretests/v3/envoy.go	`99.20% <100.00%> (+<0.01%)`	⬆️
internal/featuretests/v3/featuretests.go	`86.90% <100.00%> (+0.04%)`	⬆️
internal/provisioner/controller/gateway.go	`64.04% <100.00%> (+0.32%)`	⬆️
internal/provisioner/model/model.go	`100.00% <100.00%> (ø)`
...nternal/provisioner/objects/dataplane/dataplane.go	`85.78% <100.00%> (+0.03%)`	⬆️
... and 4 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-01-09T00:25:59Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2025-01-28T00:25:39Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2025-02-27T00:26:42Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

sunjayBhatia

we'll have to add the new omEnforcedHealthListener config file param here:

contour/site/content/docs/main/configuration.md

Line 66 in 74c0c31

## Configuration File

lets also add documentation on the new bootstrap flag here as well:

contour/site/content/docs/main/configuration.md

Line 503 in 74c0c31

### Bootstrap Flags

site/content/docs/main/config/api-reference.html

apis/projectcontour/v1alpha1/contourconfig.go

pkg/config/parameters.go

apis/projectcontour/v1alpha1/contourconfig_helpers.go

changelogs/unreleased/6794-seth-epps-minor.md

test/e2e/bootstrap/bootstrap_test.go

sunjayBhatia · 2025-04-29T20:12:43Z

Just messing around with the new health listener, set the overload manager max downstream connections to 1, port-forwarded to the new listener and sent some requests to it in quick succession, could see that the first goes through just fine and a second gets it's connection closed

I also see the failed_updates stat continually increasing, presumably from the readiness checks going to the normal health listener and requests to the admin listener which are both ignoring the global connection limit

sunjayBhatia · 2025-04-29T20:49:57Z

Just messing around with the new health listener, set the overload manager max downstream connections to 1, port-forwarded to the new listener and sent some requests to it in quick succession, could see that the first goes through just fine and a second gets it's connection closed

I also see the failed_updates stat continually increasing, presumably from the readiness checks going to the normal health listener and requests to the admin listener which are both ignoring the global connection limit

verified the same with the http ingress listener as well

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

Signed-off-by: Sunjay Bhatia <sunjay.bhatia@broadcom.com>

sunjayBhatia

Thanks for the changes covering the gateway provisioner!

we dont yet have any API surface in the provisioner to modify readiness/liveness checks on managed contour/envoy instances, but that can be a future feature where we also allow enabling the Overload Manager enforced health endpoint with a readiness check

…projectcontour#6794) Adds bootstrap flag for global downstream connection limit If set, HTTP/HTTPS Listeners will respect this limit, connections to admin/health/metrics endpoints count toward this limit Also adds ability to configure a health listener that will respect the connection limit (existing admin/health/metrics listeners are exempted) that can be used for readiness checks that desire to take this limit into account Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

seth-epps requested a review from a team as a code owner December 6, 2024 19:27

seth-epps requested review from tsaarni and sunjayBhatia and removed request for a team December 6, 2024 19:27

sunjayBhatia requested review from a team, rajatvig and izturn and removed request for a team December 6, 2024 19:28

tsaarni added the release-note/minor A minor change that needs about a paragraph of explanation in the release notes. label Dec 6, 2024

This was referenced Dec 17, 2024

Refactor stats listener seth-epps/contour#1

Open

Overload Manager - Max Global Downstream Connections #6309

Closed

seth-epps changed the title ~~Add new bootstrap flag for connection limit~~ Bootstrap flag for connection limit overload manager resource monitor Dec 19, 2024

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 9, 2025

sunjayBhatia modified the milestone: 1.31.0 Jan 13, 2025

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 28, 2025

github-actions bot closed this Feb 27, 2025

sunjayBhatia removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 24, 2025

sunjayBhatia reopened this Apr 24, 2025

sunjayBhatia requested a review from a team April 24, 2025 19:20

seth-epps added 5 commits April 26, 2025 11:18

Add new bootstrap flag for connection limit

7195d2e

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

flag help fix

9d288b2

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

gci

be7c1ec

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

gci round2

5177a7a

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

update PR number

d6f5db9

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

🤦

fac6341

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

seth-epps force-pushed the globalconnections-overloadmanager branch from 1680909 to fac6341 Compare April 26, 2025 15:18

seth-epps added 2 commits April 26, 2025 11:29

fix after conflict

8316d7c

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

fix typo in flag

b18a2a9

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

seth-epps force-pushed the globalconnections-overloadmanager branch from ff3dd7c to b18a2a9 Compare April 26, 2025 15:29

sunjayBhatia reviewed Apr 29, 2025

View reviewed changes

seth-epps added 2 commits April 30, 2025 22:45

feedback

f3ff004

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

reset 1.28 docs

343ffd5

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

seth-epps force-pushed the globalconnections-overloadmanager branch from 177bfa2 to 343ffd5 Compare May 1, 2025 02:48

seth-epps added 3 commits April 30, 2025 22:52

regenerate with updated comments

43d16be

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

missed docstring

36196ff

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

support connection limit with gateway provisioner

7642345

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

seth-epps force-pushed the globalconnections-overloadmanager branch from 0cc886d to 7642345 Compare May 2, 2025 01:08

seth-epps and others added 5 commits May 1, 2025 21:11

make generate

5e8f3b3

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

I wish I could spell

ca0e21a

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

config docs

0d922ea

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

add example

87a0a83

Signed-off-by: Seth Epps <seth.d.epps@gmail.com>

gateway provisioner unit test

45f46d4

Signed-off-by: Sunjay Bhatia <sunjay.bhatia@broadcom.com>

sunjayBhatia approved these changes May 2, 2025

View reviewed changes

sunjayBhatia enabled auto-merge (squash) May 2, 2025 17:16

sunjayBhatia merged commit 9f3f182 into projectcontour:main May 2, 2025
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bootstrap flag for connection limit overload manager resource monitor #6794

Bootstrap flag for connection limit overload manager resource monitor #6794

Uh oh!

seth-epps commented Dec 6, 2024 •

edited

Loading

Uh oh!

codecov bot commented Dec 6, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Jan 9, 2025

Uh oh!

github-actions bot commented Jan 28, 2025

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

sunjayBhatia left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sunjayBhatia commented Apr 29, 2025

Uh oh!

sunjayBhatia commented Apr 29, 2025

Uh oh!

sunjayBhatia left a comment

Uh oh!

Uh oh!

Uh oh!

Bootstrap flag for connection limit overload manager resource monitor #6794

Bootstrap flag for connection limit overload manager resource monitor #6794

Uh oh!

Conversation

seth-epps commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Manual testing

Uh oh!

codecov bot commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jan 9, 2025

Uh oh!

github-actions bot commented Jan 28, 2025

Uh oh!

github-actions bot commented Feb 27, 2025

Uh oh!

sunjayBhatia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sunjayBhatia commented Apr 29, 2025

Uh oh!

sunjayBhatia commented Apr 29, 2025

Uh oh!

sunjayBhatia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

seth-epps commented Dec 6, 2024 •

edited

Loading

codecov bot commented Dec 6, 2024 •

edited

Loading