Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

Frequent crashes and spikes in RAM use, several critical errors in logs #3188

Closed
Closed
Frequent crashes and spikes in RAM use, several critical errors in logs#3188
@JoKeyser

Description

@JoKeyser
JoKeyser
opened on May 3, 2018

Description

Since a few days, I notice frequent disconnects to my homeserver; mainly on Riot Android, but also some on Riot desktop.

In Synapse's logs, I find several errors that may hint at the problem(s), see below. Sorry that it's a fairly long dump, but I can't judge what parts are useful. Let me know if I can check/add more.

2018-05-03 23:02:00,056 - twisted - 131 - CRITICAL - - Unexpected exception from synapse.http.client.FileBodyProducer.stopProducing
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/protocols/policies.py", line 125, in connectionLost
    self.wrappedProtocol.connectionLost(reason)
  File "/usr/lib/python2.7/dist-packages/twisted/web/_newclient.py", line 964, in dispatcher
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/web/_newclient.py", line 1634, in _connectionLost_TRANSMITTING
    self._currentRequest.stopWriting()
  File "/usr/lib/python2.7/dist-packages/twisted/web/_newclient.py", line 865, in stopWriting
    _callAppFunction(self.bodyProducer.stopProducing)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/web/_newclient.py", line 194, in _callAppFunction
    function()
  File "/usr/lib/python2.7/dist-packages/twisted/web/client.py", line 1086, in stopProducing
    self._task.stop()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/task.py", line 497, in stop
    self._checkFinish()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/task.py", line 507, in _checkFinish
    raise self._completionState
twisted.internet.task.TaskStopped:

There are a bunch of ones like this:

2018-05-03 23:29:58,103 - synapse.http.server - 191 - ERROR - GET-638- Failed handle request synapse.rest.media.v1.download_resource._async_render_GET on <synapse.rest.media.v1.download_resource.DownloadResource instance at 0x7faf9a3e1d88>: <SynapseRequest at 0x7faf0ecca710 method=GET uri=/_matrix/media/v1/download/matrix.org/jdVtvZBLTBLNCdnSXfkfOibx clientproto=HTTP/1.1 site=8008>: Traceback (most recent call last):
Failure: exceptions.Exception: Consumer asked us to stop producing
2018-05-03 23:29:58,107 - twisted - 131 - CRITICAL - GET-638- Unhandled Error
Traceback (most recent call last):
Failure: exceptions.RuntimeError: Producer was not unregistered for /_matrix/media/v1/download/matrix.org/jdVtvZBLTBLNCdnSXfkfOibx

2018-05-03 23:50:30,669 - synapse.http.server - 191 - ERROR - GET-148- Failed handle request synapse.rest.media.v1.thumbnail_resource._async_render_GET on <synapse.rest.media.v1.thumbnail_resource.ThumbnailResource instance at 0x7fc773a48a28>: <SynapseRequest at 0x7fc73817d200 method=GET uri=/_matrix/media/v1/thumbnail/matrix.org/YqvlPbTsQhGREHeVRXWsHqgL?width=36&height=36&method=crop clientproto=HTTP/1.1 site=8008>: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1442, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/synapse/http/server.py", line 165, in wrapped_request_handler
    yield request_handler(self, request)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/thumbnail_resource.py", line 80, in _async_render_GET
    width, height, method, m_type
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/thumbnail_resource.py", line 233, in _respond_remote_thumbnail
    media_info = yield self.media_repo.get_remote_media_info(server_name, media_id)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/media_repository.py", line 273, in get_remote_media_info
    server_name, media_id,
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/media_repository.py", line 323, in _get_remote_me
    server_name, media_id, file_id
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/rest/media/v1/media_repository.py", line 429, in _download_remo
    filesystem_id=file_id,
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 424, in _simple_insert
    self._simple_insert_txn, table, values,
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 320, in runInteraction
    result = yield self.runWithConnection(inner_func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 365, in runWithConnection
    inner_func, *args, **kwargs
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 307, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/usr/lib/python2.7/dist-packages/twisted/enterprise/adbapi.py", line 298, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 361, in inner_func
    return func(conn, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 316, in inner_func
    func, *args, **kwargs
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 241, in _new_transaction
    r = func(txn, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 444, in _simple_insert_txn
    txn.execute(sql, vals)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 82, in execute
    self._do_execute(self.txn.execute, sql, *args)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 112, in _do_execute
    sql, *args

2018-05-04 00:14:07,385 - synapse.federation.federation_server - 620 - ERROR - PUT-1236- Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/synapse/federation/federation_server.py", line 616, in on_edu
    yield handler(origin, content)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/handlers/receipts.py", line 84, in _received_remote_receipt
    yield self._handle_new_receipts(receipts)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/handlers/receipts.py", line 101, in _handle_new_receipts
    room_id, receipt_type, user_id, event_ids, data
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/receipts.py", line 456, in insert_receipt
    stream_id=stream_id,
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/_base.py", line 323, in runInteraction
    after_callback(*after_args, **after_kwargs)
  File "/usr/lib/python2.7/dist-packages/synapse/storage/receipts.py", line 308, in _invalidate_get_users_with_receipts_in_room
    if user_id in res:
TypeError: argument of type 'ObservableDeferred' is not iterable

2018-05-04 00:19:13,429 - synapse.federation.transport.server - 188 - ERROR - PUT-1913- authenticate_request failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/synapse/federation/transport/server.py", line 181, in new_func
    origin = yield authenticator.authenticate_request(request, content)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/federation/transport/server.py", line 145, in authenticate_request
    yield self.keyring.verify_json_for_server(origin, json_request)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/usr/lib/python2.7/dist-packages/synapse/crypto/keyring.py", line 768, in _handle_key_deferred
    Codes.UNAUTHORIZED,
SynapseError: 401: Invalid signature for server MY_DOMAIN with key ed25519:a_RBEL

2018-05-04 00:23:36,277 - twisted - 131 - CRITICAL - - Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 103, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 86, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 619, in _doReadOrWrite
    why = selectable.doWrite()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1168, in doWrite
    "doWrite called on a %s" % reflect.qual(self.__class__))
exceptions.RuntimeError: doWrite called on a twisted.internet.udp.Port

Steps to reproduce

Running Synapse... :/.

Version information

If not matrix.org:

  • Version: Synapse/0.27.2 and 0.28.1.
  • Install method: Debian's apt.
  • Platform: Debian buster, amd64, on dedicated hardware with 4 cores and 4 GiB, http://www.pcengines.ch/apu2c4.htm.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions