This repository was archived by the owner on Apr 26, 2024. It is now read-only.
-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
random 500 failures with exceptions.ValueError: Counters can only be incremented by non-negative amounts.
#3548
Copy link
Copy link
Closed
Labels
Description
from time to time random requests fail with a 500 error. it looks like we're trying to report a negative CPU usage.
2018-07-16 14:09:14,467 - synapse.http.server - 101 - ERROR - POST-2245 - Failed handle request via <function _async_render at 0x7f78a28f5de8>: <SynapseRequest at 0x7f7884056998 method='POST' uri='/_matrix/client/r0/createRoom?access_token=<redacted>' clientproto='HTTP/1.1' site=27060>: Traceback (most recent call last):
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1442, in gotResult
_inlineCallbacks(r, g, deferred)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/python/failure.py, line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/http/server.py, line 78, in wrapped_request_handler
yield h(self, request)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/python/failure.py, line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/http/server.py, line 304, in _async_render
callback_return = yield callback(request, **kwargs)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/python/failure.py, line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/rest/client/v1/room.py, line 73, in on_POST
requester, self.get_room_config(request)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/python/failure.py, line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/handlers/room.py, line 198, in create_room
creator_join_profile=creator_join_profile,
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/python/failure.py, line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/handlers/room.py, line 380, in _send_events_for_new_room
content={history_visibility: config[history_visibility]}
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/python/failure.py, line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/handlers/room.py, line 297, in send
ratelimit=False
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/python/failure.py, line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/handlers/message.py, line 669, in create_and_send_nonmember_event
txn_id=txn_id
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/python/failure.py, line 393, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/handlers/message.py, line 503, in create_event
prev_events_and_hashes=prev_events_and_hashes,
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py, line 1386, in _inlineCallbacks
result = g.send(result)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/util/metrics.py, line 55, in measured_func
r = yield func(self, *args, **kwargs)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/synapse/synapse/util/metrics.py, line 108, in __exit__
block_ru_utime.labels(self.name).inc(usage.ru_utime)
File /var/lib/jenkins/jobs/Sytest-Postgres-Branches/workspace/.tox/py27/local/lib/python2.7/site-packages/prometheus_client/core.py, line 654, in inc
raise ValueError('Counters can only be incremented by non-negative amounts.')
exceptions.ValueError: Counters can only be incremented by non-negative amounts.