Skip to content

Conversation

mergeconflict
Copy link
Member

Description: Reintroduce dispatcher stats previously reverted in #6649. Dispatcher stats are now gated on a new bootstrap config parameter, enable_dispatcher_stats, and disabled by default.
Risk Level: Low
Testing: Manually verified that config is respected (see #6582, this still requires integration tests).
Docs Changes: Included a note about how statsd doesn't play well with these high-volume histograms.

Signed-off-by: Dan Rosen mergeconflict@google.com

Signed-off-by: Dan Rosen <mergeconflict@google.com>
@mergeconflict
Copy link
Member Author

/review @htuch @mattklein123

Copy link
Member Author

@mergeconflict mergeconflict left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few notes for reviewers inline...

WorkerFactory& worker_factory)
: server_(server), factory_(listener_factory), stats_(generateStats(server.stats())),
WorkerFactory& worker_factory,
bool enable_dispatcher_stats)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: I put this here, rather than as a param in startWorkers() or a new initializeStats() method, to avoid touching the ListenerManager interface. I could do this differently if anybody feels I should...

@@ -90,6 +96,14 @@ class DispatcherImplTest : public testing::Test {
TimerPtr keepalive_timer_;
};

// TODO(mergeconflict): We also need integration testing to validate that the expected histograms
// are written when `enable_dispatcher_stats` is true. See issue #6582.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm planning to implement integration tests in a following PR, once I've dealt with #6582, if that's still ok.

running---but if this number elevates substantially above its normal observed baseline, it likely
indicates kernel scheduler delays.

These statistics can be enabled by setting :ref:`enable_dispatcher_stats <envoy_api_field_config.bootstrap.v2.Bootstrap.enable_dispatcher_stats>`
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bit is new.

// Enable :ref:`stats for event dispatcher <operations_performance>`, defaults to false.
// Note that this records a value for each iteration of the event loop on every thread. This
// should normally be minimal overhead, but when using
// :ref:`statsd <envoy_api_msg_config.metrics.v2.StatsdSink>`, it will send each observed value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it just statsd or also other stats sinks?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, just statsd. The Monarch sink appears to just dump histograms on the floor, and the Hystrix and Metrics sinks both do the right thing with parent histograms.

@mergeconflict
Copy link
Member Author

/retest

@repokitteh-read-only
Copy link

🔨 rebuilding ci/circleci: release (failed build)

🐱

Caused by: a #6659 (comment) was created by @mergeconflict.

see: more, trace.

htuch
htuch previously approved these changes Apr 22, 2019
Copy link
Member

@htuch htuch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM; ideally when you add new behavioral tests these also cover the enable/disable case.

Signed-off-by: Dan Rosen <mergeconflict@google.com>
@htuch
Copy link
Member

htuch commented Apr 22, 2019

@mergeconflict needs master merge for version history.

Copy link
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks.

@htuch htuch merged commit 481a127 into envoyproxy:master Apr 23, 2019
mpuncel added a commit to mpuncel/envoy that referenced this pull request Apr 24, 2019
* master:
  docs: add extension policy (envoyproxy#6678)
  ext_authz: added ability to detect partial request body data (envoyproxy#6583)
  version_history.rst: jwt_authn change missed 1.10.0 (envoyproxy#6684)
  docs: fix link in pull request template (envoyproxy#6679)
  Explicitly convert absl::string_view to std::string. (envoyproxy#6687)
  docs: improving watermark docs/comments (envoyproxy#6683)
  http filter: add CSRF filter (envoyproxy#6470)
  event: reintroduce dispatcher stats (envoyproxy#6659)
  security: postmortem for CVE-2019-990[01] (envoyproxy#6597)
  Improve build rules for (test only) library quic_port_utils. (envoyproxy#6672)
  spell check: skip unsupported extensions when called with a file (envoyproxy#6648)
  Changed TestHooks to ListenerHooks (envoyproxy#6642)
  proto: move extension-specific linking validation into extensions (envoyproxy#6657)
  stats: add/test heterogenous set of StatNameStorage objects. (envoyproxy#6504)
  docs: move xds protocol to rst (envoyproxy#6670)
  fix version history order (envoyproxy#6671)

Signed-off-by: Michael Puncel <mpuncel@squareup.com>
@mergeconflict mergeconflict deleted the enable_dispatcher_stats branch April 25, 2019 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants