This repository was archived by the owner on Apr 26, 2024. It is now read-only.
-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
trial
test runs leak memory #15622
Copy link
Copy link
Closed
Labels
O-FrequentAffects or can be seen by most users regularly or impacts most users' first experienceAffects or can be seen by most users regularly or impacts most users' first experienceS-MinorBlocks non-critical functionality, workarounds exist.Blocks non-critical functionality, workarounds exist.T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.Bugs, crashes, hangs, security vulnerabilities, or other reported issues.Z-Dev-WishlistMakes developers' lives better, but doesn't have direct user impactMakes developers' lives better, but doesn't have direct user impact
Description
#15334 introduced a bug where HomeServer
instances no longer get garbage collected during trial
test runs. As a result, memory usage grows to multiple gigabytes during test runs.
See https://github.com/matrix-org/synapse/pull/15334/files#diff-f50ec854a65082453f4086048f20c291ca6020fab973486a9da2285025394cc8R180.
When we garbage collect generation 0, self.hs
still holds a reference to the HomeServer
, which gets promoted to generation 1 and then never collected.
Apart from that, there are also a couple of other leaks at play.
- Fix
HomeServer
s leaking due to not being GCed.
FixHomeServer
s leaking duringtrial
test runs #15630 - Fix
FederationRateLimiter
(and referencedHomeServer
s) leaking due to_rate_limiter_instances
global.
FixHomeServer
s leaking duringtrial
test runs #15630 - Fix config cache added in Speedup tests by caching HomeServerConfig instances #15284 growing too large (~360 entries).
HomeServerConfig
s are pretty heavy and hold on to a lot of stuff, including Jinja templates. A suitable max size for the cache might be 8-16 entries.
Limit the size of theHomeServerConfig
cache in trial test runs #15646 - Fix
HomeServer
s leaking via synapse.util.caches.caches_by_name. - Fix
HomeServer
s leaking via synapse.util.caches.CACHE_METRIC_REGISTRY. - Fix
HomeServer
s leaking via synapse.config.cache._CACHES.
The last 3 are fairly annoying to fix, so I've left them alone.
Metadata
Metadata
Assignees
Labels
O-FrequentAffects or can be seen by most users regularly or impacts most users' first experienceAffects or can be seen by most users regularly or impacts most users' first experienceS-MinorBlocks non-critical functionality, workarounds exist.Blocks non-critical functionality, workarounds exist.T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.Bugs, crashes, hangs, security vulnerabilities, or other reported issues.Z-Dev-WishlistMakes developers' lives better, but doesn't have direct user impactMakes developers' lives better, but doesn't have direct user impact