Skip to content

Conversation

danielhochman
Copy link
Contributor

@danielhochman danielhochman commented Aug 3, 2018

Signed-off-by: Daniel Hochman danielhochman@users.noreply.github.com

This reverts commit 028387a, #3938.

After this PR was deployed to production, our instances would periodically experience a sharp and sustained uptick in CPU.

screenshot from 2018-08-03 10-20-36

I removed traffic from one of the instances. It continued to consume 50% CPU. I took a perf capture to get the call stack which pointed me to this PR in the diff.

image

For fun I also generated a flame graph. flame graph

This reverts commit 028387a.

Signed-off-by: Daniel Hochman <danielhochman@users.noreply.github.com>
Copy link
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging on this.

@mattklein123 mattklein123 merged commit 6a1868d into master Aug 3, 2018
@mattklein123 mattklein123 deleted the revert-3938 branch August 3, 2018 19:00
zuercher added a commit to turbinelabs/envoy that referenced this pull request Aug 6, 2018
Converts TcpProxy::Filter and WebSocket::WsHandlerImpl to use
Tcp::ConnectionPool to obtain connections. Much of the stats
handling and connection timeouts are handled by the connection
pool.

Stats were manually verified by comparing stats produced by
the tcp_proxy_integration_test with and without the connection
pool change.

This reapplies an earlier changed reverted due to envoyproxy#4043. That
bug is fixed by performing the connection pools state management
before forwarding callbacks.

*Risk Level*: medium
*Testing*: unit/integration testing
*Docs Changes*: n/a
*Release Notes*: n/a
*Fixes*: envoyproxy#4043

Signed-off-by: Stephan Zuercher <stephan@turbinelabs.io>
zuercher added a commit to turbinelabs/envoy that referenced this pull request Aug 6, 2018
…arding

callbacks and tracking whether the connection has been invalidated
or not (allowing the callback to use the connection to e.g., obtain
a dispatcher).

Signed-off-by: Stephan Zuercher <stephan@turbinelabs.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants