I thought that the JVM args used in Test tasks used org.gradle.jvmargs by default 🤔. I've fixed OOM in other projects by increasing Xmx in org.gradle.jvmargs. I'll do some testing...

UPDATE: I did some testing and I can see in the info log that this PR does correctly increase the Xmx from the default, which is too low. https://github.com/gradle/gradle/blob/6a61e6d812b2cefb7fb529f62ab4c867ca80c1d7/platforms/core-runtime/launcher/src/main/java/org/gradle/launcher/daemon/configuration/DaemonParameters.java#L47

On GitHub CI Xmx is set to 12g - maybe we should reduce that to allow more memory for the tests?

Also, I think that setting MaxMetaspaceSize is a bit out of fashion these days - setting Xmx should be enough, and then Java will pick a sensible default for MaxMetaspaceSize, as a percentage of Xmx.

I did not check how the JVM would actually do allocations (grab configured limits from the OS instantly or increase up to configured limits). I also did not check what might be required. So I have no better insights than you on this topic. Would it be OK if you'd just adjust the setting as you see fit (as part of this PR or in a separate one)?

I was hoping you had more expertise 😅. I've pushed an update that I think should work better though, so let's see...