Skip to content

Akka.Remote - Handling the case when remote node abruptly goes down (#2460) #2466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 20, 2017
Merged

Akka.Remote - Handling the case when remote node abruptly goes down (#2460) #2466

merged 1 commit into from
Jan 20, 2017

Conversation

maxcherednik
Copy link
Contributor

@maxcherednik maxcherednik commented Jan 20, 2017 •
edited
Loading

A bit more logging for the known cases

maxcherednik
maxcherednik commented Jan 20, 2017
View reviewed changes
src/core/Akka.Remote/Transport/Helios/HeliosTcpTransport.cs
var heliosException = (ConnectException) e.InnerException;
var socketException = heliosException?.InnerException as SocketException;
var heliosException = (ConnectException)e.InnerException;
var socketException = heliosException.InnerException as SocketException;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary null check removed

maxcherednik
maxcherednik commented Jan 20, 2017
View reviewed changes
src/core/Akka.Remote/Transport/Helios/HeliosTcpTransport.cs
@@ -290,13 +301,13 @@ public HeliosTcpTransport(ActorSystem system, Config config)

var associate = await clientBootstrap.ConnectAsync(socketAddress);

var handler = (TcpClientHandler) associate.Pipeline.Last();
var handler = (TcpClientHandler)associate.Pipeline.Last();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about this, Ctrl +K,D is my favorite. If it's forbidden, I will stop doing this....

maxcherednik
maxcherednik commented Jan 20, 2017
View reviewed changes
src/core/Akka.Remote/Transport/Helios/HeliosTcpTransport.cs

NotifyListener(new Disassociated(DisassociateInfo.Shutdown));
}
else if (se?.SocketErrorCode == SocketError.ConnectionReset)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New case is here. When it happens, we log the warning and sending DisassociateInfo.Shutdown.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@maxcherednik maxcherednik changed the title Handling the case when remote node abruptly goes down (#2460) Akka.Remote - Handling the case when remote node abruptly goes down (#2460) Jan 20, 2017
@maxcherednik
Copy link
Contributor Author

New logging:

2017-01-20 12:15:14.344 [7] [(null)] INFO  Akka.Event.DummyClassForStringSources - Starting remoting
2017-01-20 12:15:14.454 [7] [(null)] INFO  Akka.Event.DummyClassForStringSources - Remoting started; listening on addresses : [akka.tcp://riskengine@127.0.0.1:4054]
2017-01-20 12:15:14.456 [4] [(null)] INFO  Akka.Event.DummyClassForStringSources - Remoting now listens on addresses: [akka.tcp://riskengine@127.0.0.1:4054]
2017-01-20 12:15:14.516 [6] [(null)] INFO  Akka.Event.DummyClassForStringSources - Cluster Node [akka.tcp://riskengine@127.0.0.1:4054] - Starting up...
2017-01-20 12:15:14.536 [7] [(null)] INFO  Akka.Event.DummyClassForStringSources - Cluster Node [akka.tcp://riskengine@127.0.0.1:4054] - Started up successfully
2017-01-20 12:15:14.544 [1] [(null)] INFO  Akka.WidgetHolder.WdgetHolderService - Started
2017-01-20 12:15:25.834 [4] [(null)] INFO  Akka.Cluster.ClusterCoreDaemon - Welcome from [akka.tcp://riskengine@127.0.0.1:4053]
2017-01-20 12:17:28.374 [4] [(null)] WARN  Akka.Remote.Transport.Helios.TcpClientHandler - Connection was reset by the remote peer. Channel [[::ffff:127.0.0.1]:62777->[::ffff:127.0.0.1]:4053](Id=ChannelId(-1125123200))
2017-01-20 12:17:28.380 [21] [(null)] WARN  Akka.Remote.Transport.Helios.TcpClientHandler - Connection was reset by the remote peer. Channel [[::ffff:127.0.0.1]:62777->[::ffff:127.0.0.1]:4053](Id=ChannelId(-1125123200))
2017-01-20 12:17:29.065 [19] [(null)] INFO  Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef - Message Heartbeat from akka://riskengine/system/cluster/core/daemon/heartbeatSender to akka://riskengine/deadLetters was not delivered. 1 dead letters encountered.
2017-01-20 12:17:29.823 [4] [(null)] INFO  Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef - Message GossipStatus from akka://riskengine/system/cluster/core/daemon to akka://riskengine/deadLetters was not delivered. 2 dead letters encountered.
2017-01-20 12:17:30.072 [7] [(null)] INFO  Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef - Message Heartbeat from akka://riskengine/system/cluster/core/daemon/heartbeatSender to akka://riskengine/deadLetters was not delivered. 3 dead letters encountered.
2017-01-20 12:17:31.082 [4] [(null)] INFO  Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef - Message Heartbeat from akka://riskengine/system/cluster/core/daemon/heartbeatSender to akka://riskengine/deadLetters was not delivered. 4 dead letters encountered.
2017-01-20 12:17:32.093 [4] [(null)] INFO  Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef - Message Heartbeat from akka://riskengine/system/cluster/core/daemon/heartbeatSender to akka://riskengine/deadLetters was not delivered. 5 dead letters encountered.
2017-01-20 12:17:32.983 [20] [(null)] WARN  Akka.Cluster.ClusterCoreDaemon - Cluster Node [akka.tcp://riskengine@127.0.0.1:4054] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://riskengine@127.0.0.1:4053, status = Up, role=[riskengine], upNumber=1)]. Node roles [riskenginewidget]
2017-01-20 12:17:33.102 [5] [(null)] INFO  Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef - Message Heartbeat from akka://riskengine/system/cluster/core/daemon/heartbeatSender to akka://riskengine/deadLetters was not delivered. 6 dead letters encountered.
2017-01-20 12:17:35.139 [6] [(null)] WARN  Akka.Remote.EndpointWriter - AssociationError [akka.tcp://riskengine@127.0.0.1:4054] -> akka.tcp://riskengine@127.0.0.1:4053: Error [No connection could be made because the target machine actively refused it tcp://riskengine@127.0.0.1:4053] []
2017-01-20 12:17:35.143 [21] [(null)] WARN  Akka.Event.DummyClassForStringSources - Tried to associate with unreachable remote address [akka.tcp://riskengine@127.0.0.1:4053]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [No connection could be made because the target machine actively refused it tcp://riskengine@127.0.0.1:4053] 
2017-01-20 12:17:35.144 [21] [(null)] INFO  Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef - Message Heartbeat from akka://riskengine/system/cluster/core/daemon/heartbeatSender to akka://riskengine/deadLetters was not delivered. 7 dead letters encountered.

Aaronontheweb
Aaronontheweb approved these changes Jan 20, 2017
View reviewed changes
Copy link
Member

@Aaronontheweb Aaronontheweb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

src/core/Akka.Remote/Transport/Helios/HeliosTcpTransport.cs

NotifyListener(new Disassociated(DisassociateInfo.Shutdown));
}
else if (se?.SocketErrorCode == SocketError.ConnectionReset)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@Aaronontheweb
Copy link
Member

Resolves #2460

@Aaronontheweb Aaronontheweb merged commit 7d05d5b into akkadotnet:dev Jan 20, 2017
@maxcherednik maxcherednik deleted the akka_remote_logging branch August 2, 2017 20:11
maxcherednik added a commit to maxcherednik/akka.net that referenced this pull request Aug 2, 2017
@maxcherednik
Took the exact method from the 1.1.3 old pull-request (akkadotnet#2466)…
2f263b5 
… Issue: (akkadotnet#2754)
maxcherednik added a commit to maxcherednik/akka.net that referenced this pull request Aug 2, 2017
@maxcherednik
Took the exact method from the 1.1.3 old pull-request (akkadotnet#2466)…
bcee3c6 
… Issue: (akkadotnet#2754)
maxcherednik added a commit to maxcherednik/akka.net that referenced this pull request Aug 3, 2017
@maxcherednik
Took the exact method from the 1.1.3 old pull-request (akkadotnet#2466)…
84cd1f8 
… Issue: (akkadotnet#2754)
@maxcherednik maxcherednik mentioned this pull request Aug 3, 2017
Merged
Aaronontheweb pushed a commit that referenced this pull request Aug 4, 2017
@maxcherednik @Aaronontheweb
Took the exact method from the 1.1.3 old pull-request (#2466) Issue: (#…
286e356 
…2754) (#2923)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Aaronontheweb Aaronontheweb Aaronontheweb approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@maxcherednik @Aaronontheweb