We have found the cause of the big perf loss on Akka.Remote. this turned out not to have anything to do with Helios at all, but rather the recent introduction of the ForkJoinDispatcher.

The ForkJoinDispatcher uses the Helios DedicatedThreadPool, which internally uses a BlockingQueue for buffering things to execute.
The blocking queue is just too expensive, it uses locking primitives that cause too much latency.

We need a better way to handle this.
One thing we could try is to port the real fork join thread pool from JVM Akka.
I have no idea how much work that would be but it is one way forward and it would be interesting to see the result.

The reason this was introduced was that we saw a lot of noisy neighbor problems with heartbeats in Akka.Cluster.

However, heartbeats now run on the DedicatedThreadScheduler which is not the same as the above.
So heartbeats are not dependant on the DedicatedThreadPool.