-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Description
Coordinated shutdowns ClusterLeave phase only allows those members to leave cluster whose status is UP.
But in short running application where WeaklyUp members are allowed to join cluster, members might want to leave cluster while in WeaklyUp status.
Use case is as follows:
- Application joins existing akka cluster
- Does some work
- Leaves the cluster
But at step 3, the status of member was still (Joining or WeaklyUp) and if you try to run CoordinatedShutdown in this case, many of the phases ClusterLeave and phases after that times out .
Snippet from ClusterDaemon.scala which is invoked on Running ClusterLeave phase
def leaving(address: Address): Unit = { // only try to update if the node is available (in the member ring) if (latestGossip.members.exists(m ⇒ m.address == address && m.status == Up)) { val newMembers = latestGossip.members map { m ⇒ if (m.address == address) m.copy(status = Leaving) else m } // mark node as LEAVING val newGossip = latestGossip copy (members = newMembers) updateLatestGossip(newGossip) logInfo("Marked address [{}] as [{}]", address, Leaving) publishMembershipState() // immediate gossip to speed up the leaving process gossip() } }
can this check be like
if (latestGossip.members.exists(m ⇒ m.address == address && (m.status == Up || m.status == WeaklyUp)))
or another solution would be to use counter and wait for member to become up.
For time being as a workaround we have implemented something like this CswCoordinatedShutdown.scala