I tracked down a PermGen leak in sbt, which can easily be worked around (although other PermGen leaks may exist).

The full story is below, but in short object finalizers are not called frequently enough by the VM (in jdk 6 and 7), causing classloaders to survive long past their time.

The solution is adding after the parts of runTask that create a new classloader, in order to launch Scala or other utilities, the sequence:

{ System.gc(); java.lang.System.runFinalization(); System.gc() }

That must be done from an enclosing stack frame that no longer contains any reference, direct or indirect, to the generated classloader (note that escape analysis was not generally on by default in jdk 6).

The complete, albeit condensed explanation, is:

• Each time sbt invokes the Scala compiler, by creating a new classloader, the classloader is not released.
• the classloader is kept in the heap by means of a chain of links involving weak references. In particular, when the garbage collector runs and frees the referent from a weak reference, the referred instance may still being tracked if some finalization is still pending.
• if finalization is not run promptly, the classloader cannot be freed, and its permgen cannot be released.
• file descriptors, like FileInputStream or JarFile, are all finalizable, and sbt uses a lot of them.
• the jdk 6 and 7 are quite lazy when running finalizers. If there is still heap space, they will not rush to complete finalization in order to free memory. The VMs will not notice that PermGen is slowly running out.
• as a net result, I saw cases in which there are more than 47'000 file descriptors lying idle in the system, indirectly keeping classloaders from being reclaimed, and leading some code to crash despite being given 1GB of PermGen.
• adding an explicit invocation to runFinalization(), following the exact sequence described above, will solve the problem. The first gc() causes the referent of the WeakReferences to be released, and tracks the newly freed finalizable objects. The runFinalization() call forces the jvm to deal with those instances immediately. Finally, the second gc() will deallocate them. Once the file descriptors are freed, the weak references are also freed, the classloader is freed, and therefore its PermGen space is in the end also freed.

More in general, when using together weak references, multiple classloaders, and files (even if you close them), manual invocations to runFinalization() will be needed, or you will get leaks that may show up as "Too many files" or "PermGen Error".

Java 8 replaces the PermGen with a "Meta" space, but it is still subject to leaks. However, it calls finalizers more promptly, which avoids this issue. Java 6 and Java 7 are both affected.