Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

TypeError: Object of type FrozenEvent is not JSON serializable #11772

@anoadragon453

Description

@anoadragon453

When testing out commit d1e6333 on matrix.org today, we ended up with the following stacktrace:

stacktrace
2022-01-19 13:31:19,225 - twisted - 279 - CRITICAL - sentinel - Unhandled error in Deferred:
2022-01-19 13:31:19,237 - twisted - 279 - CRITICAL - sentinel - 
Capture point (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/generic_worker.py", line 514, in <module>
    main()
  File "/home/synapse/src/synapse/app/generic_worker.py", line 510, in main
    start(sys.argv[1:])
  File "/home/synapse/src/synapse/app/generic_worker.py", line 505, in start
    _base.start_worker_reactor("synapse-generic-worker", config)
  File "/home/synapse/src/synapse/app/_base.py", line 126, in start_worker_reactor
    run_command=run_command,
  File "/home/synapse/src/synapse/app/_base.py", line 179, in start_reactor
  File "/home/synapse/src/synapse/app/_base.py", line 163, in run
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1318, in run
    self.mainLoop()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1328, in mainLoop
    reactorBaseSelf.runUntilCurrent()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 967, in runUntilCurrent
    f(*a, **kw)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 701, in errback
    self._startRunCallbacks(fail)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 859, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1751, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
Traceback (most recent call last):
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1658, in _inlineCallbacks
    cast(Failure, result).throwExceptionIntoGenerator, gen
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/synapse/src/synapse/http/server.py", line 779, in _async_write_json_to_request_in_thread
    json_str = await defer_to_thread(request.reactor, encode, span)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/threadpool.py", line 255, in <lambda>
    ctx, func, *args, **kw
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/home/synapse/src/synapse/logging/context.py", line 958, in g
    return f(*args, **kwargs)
  File "/home/synapse/src/synapse/http/server.py", line 772, in encode
    res = json_encoder(json_object)
  File "/home/synapse/src/synapse/http/server.py", line 663, in _encode_json_bytes
    return json_encoder.encode(json_object).encode("utf-8")
  File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/synapse/src/synapse/util/__init__.py", line 67, in _handle_frozendict
    "Object of type %s is not JSON serializable" % obj.__class__.__name__
TypeError: Object of type FrozenEvent is not JSON serializable

Sentry link for those with access: https://sentry.matrix.org/sentry/synapse-matrixorg/issues/239798/

This issue only occurs on our client_reader workers, but does occur frequently across multiple instances of it.

Unfortunately no surrounding processed request lines are relevant. However, there do seem to be a number of replication related lines, so that may be a clue...

Note that a similar error has occurred before: #8678.

Metadata

Metadata

Assignees

Labels

T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.X-RegressionSomething broke which worked on a previous releaseX-Release-BlockerMust be resolved before making a release

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions