Akka incorrectly schedules continuations after .Ask, causing deadlocks and/or delays #2546
Description
Edit3: fixed the repro and root cause, with possible fix
The way Akka currently schedules continuations on Ask (and probably other places), user code runs on (and blocks) the Helios worker thread.
For one repro, see https://github.com/akkadotnet/akka.net/pull/2548/files , but it is NOT directly related to .Result, doing anything after await Ask locks up the system for the duration.
One possible fix would be to construct the TaskCompletionSource with TaskCreationOptions.RunContinuationsAsynchronously. That way, the continuations (the user code) is executed on the thread pool, freeing the Helios worker thread.
see the callstack at this breakpoint:
> AkkaDeadlockTest.exe!AkkaDeadlockTest.MainWindow.Button_Click_2(object sender, System.Windows.RoutedEventArgs e) Line 21 C#
[Resuming Async Method]
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine) Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run() Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0() Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke() Unknown
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0() Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke() Unknown
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask) Unknown
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations() Unknown
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Unknown
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Unknown
mscorlib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.TrySetResult(System.__Canon result) Unknown
mscorlib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.SetResult(System.__Canon result) Unknown
Akka.dll!Akka.Util.Internal.TaskExtensions.CastTask.AnonymousMethod__0(System.Threading.Tasks.Task<object> _) Unknown
mscorlib.dll!System.Threading.Tasks.ContinuationTaskFromResultTask<object>.InnerInvoke() Unknown
mscorlib.dll!System.Threading.Tasks.Task.Execute() Unknown
mscorlib.dll!System.Threading.Tasks.Task.ExecutionContextCallback(object obj) Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot) Unknown
mscorlib.dll!System.Threading.Tasks.Task.ExecuteEntry(bool bPreventDoubleExecution) Unknown
mscorlib.dll!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task task, bool taskWasPreviouslyQueued) Unknown
mscorlib.dll!System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task task, bool taskWasPreviouslyQueued) Unknown
mscorlib.dll!System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(System.Threading.Tasks.Task task, bool needsProtection) Unknown
mscorlib.dll!System.Threading.Tasks.StandardTaskContinuation.Run(System.Threading.Tasks.Task completedTask, bool bCanInlineContinuationTask) Unknown
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations() Unknown
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Unknown
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Unknown
mscorlib.dll!System.Threading.Tasks.TaskCompletionSource<System.__Canon>.TrySetResult(System.__Canon result) Unknown
Akka.dll!Akka.Actor.FutureActorRef.TellInternal(object message, Akka.Actor.IActorRef sender) Unknown
Akka.dll!Akka.Actor.ActorRefBase.Tell(object message, Akka.Actor.IActorRef sender) Unknown
Akka.Remote.dll!Akka.Remote.DefaultMessageDispatcher.Dispatch(Akka.Actor.IInternalActorRef recipient, Akka.Actor.Address recipientAddress, SerializedMessage message, Akka.Actor.IActorRef senderOption) Unknown
Akka.Remote.dll!Akka.Remote.EndpointReader.Reading.AnonymousMethod__11_1(Akka.Remote.Transport.InboundPayload inbound) Unknown
[Lightweight Function]
Akka.dll!Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture<System.__Canon, System.__Canon, System.__Canon, System.__Canon>.Handle(System.__Canon value) Unknown
Akka.dll!Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(object message, Akka.Tools.MatchHandler.PartialAction<object> partialAction) Unknown
Akka.dll!Akka.Actor.ReceiveActor.OnReceive(object message) Unknown
Akka.dll!Akka.Actor.UntypedActor.Receive(object message) Unknown
Akka.dll!Akka.Actor.ActorBase.AroundReceive(Akka.Actor.Receive receive, object message) Unknown
Akka.dll!Akka.Actor.ActorCell.ReceiveMessage(object message) Unknown
Akka.dll!Akka.Actor.ActorCell.Invoke(Akka.Actor.Envelope envelope) Unknown
Akka.dll!Akka.Dispatch.Mailbox.ProcessMailbox(int left, long deadlineTicks) Unknown
Akka.dll!Akka.Dispatch.Mailbox.ProcessMailbox() Unknown
Akka.dll!Akka.Dispatch.Mailbox.Run.AnonymousMethod__36_0() Unknown
Akka.dll!Akka.Actor.ActorCell.UseThreadContext(System.Action action) Unknown
Akka.dll!Akka.Dispatch.Mailbox.Run() Unknown
Akka.dll!Helios.Concurrency.DedicatedThreadPool.PoolWorker.RunThread() Unknown
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() Unknown
At the bottom is Akka.dll!Helios.Concurrency.DedicatedThreadPool.PoolWorker.RunThread so this is the helios worker.
At the top is my user code AFTER the Ask. This code SHOULD be run on the ThreadPool, it doesn't make sense to run this on the worker thread, blocking it. As a consumer of Akka, I don't have any real control over that, so it needs to be fixed in Akka.
Version:
Akka =1.1.3.