Skip to content

Coordinated Shutdown leave phase times out if member status is WeaklyUp #24140

@kpritam

Description

@kpritam

Coordinated shutdowns ClusterLeave phase only allows those members to leave cluster whose status is UP.

But in short running application where WeaklyUp members are allowed to join cluster, members might want to leave cluster while in WeaklyUp status.

Use case is as follows:

  1. Application joins existing akka cluster
  2. Does some work
  3. Leaves the cluster

But at step 3, the status of member was still (Joining or WeaklyUp) and if you try to run CoordinatedShutdown in this case, many of the phases ClusterLeave and phases after that times out .

Snippet from ClusterDaemon.scala which is invoked on Running ClusterLeave phase

def leaving(address: Address): Unit = { // only try to update if the node is available (in the member ring) if (latestGossip.members.exists(m ⇒ m.address == address && m.status == Up)) { val newMembers = latestGossip.members map { m ⇒ if (m.address == address) m.copy(status = Leaving) else m } // mark node as LEAVING val newGossip = latestGossip copy (members = newMembers) updateLatestGossip(newGossip) logInfo("Marked address [{}] as [{}]", address, Leaving) publishMembershipState() // immediate gossip to speed up the leaving process gossip() } }

can this check be like
if (latestGossip.members.exists(m ⇒ m.address == address && (m.status == Up || m.status == WeaklyUp)))

or another solution would be to use counter and wait for member to become up.

For time being as a workaround we have implemented something like this CswCoordinatedShutdown.scala

Metadata

Metadata

Assignees

No one assigned

    Labels

    1 - triagedTickets that are safe to pick up for contributing in terms of likeliness of being acceptedbughelp wantedIssues that the core team will likely not have time to work ont:cluster

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions