Cluster singleton manager: don't send member events to FSM during shutdown #24236
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tdown There exists a race where a cluter node that is being downed seens its self as the oldest node (as it has had the other nodes removed) and it takes over the singleton manager sending the real oldest node to go into the End state meaning that cluster singletons never work again. This fix simply prevents Member events being given to the Cluster Manager FSM during a shut down, instread relying on SelfExiting. This also hardens the test by not downing the node that the current sharding coordinator is running on as well as fixing a bug in the probes.
|
The test fails when the downed node has the other remembers removed. This happens locally 1/10ish. |
|
Gong to run the multi node jobs on repeat job for this |
|
Test PASSed. |
patriknw
reviewed
Jan 3, 2018
There was a problem hiding this comment.
good catch
wonder if the original issue should be kept and this is another thing?
I wrote:
The real issue that should be fixed is that there seems to be a race between the CS and the ClusterSingleton observing OldestChanged and terminating coordinator singleton before the graceful sharding stop is done
|
Yes as i don't think it'll fix that one ^ this just fixed one test issue + one actual bug. So lets keep #24113 open for changing the test back to shutting down a node that has the coordinator |
johanandren
approved these changes
Jan 4, 2018
|
This one causing a lot of failures, someone else from @akka/akka-team mind reviewing this? |
raboof
approved these changes
Jan 5, 2018
| @@ -409,7 +410,7 @@ class Cluster(val system: ExtendedActorSystem) extends Extension { | |||
| * Should not called by the user. The user can issue a LEAVE command which will tell the node | |||
| * to go through graceful handoff process `LEAVE -> EXITING -> REMOVED -> SHUTDOWN`. | |||
| */ | |||
| private[cluster] def shutdown(): Unit = { | |||
| @InternalApi private[cluster] def shutdown(): Unit = { | |||
manonthegithub
pushed a commit
to manonthegithub/akka
that referenced
this pull request
Jan 31, 2018
…tdown (akka#24236) There exists a race where a cluter node that is being downed seens its self as the oldest node (as it has had the other nodes removed) and it takes over the singleton manager sending the real oldest node to go into the End state meaning that cluster singletons never work again. This fix simply prevents Member events being given to the Cluster Manager FSM during a shut down, instread relying on SelfExiting. This also hardens the test by not downing the node that the current sharding coordinator is running on as well as fixing a bug in the probes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There exists a race where a cluster node that is being downed sees its
self as the oldest node (as it has had the other nodes removed) and it
takes over the singleton manager sending the real oldest node to go into
the End state meaning that cluster singletons never work again.
This fix simply prevents Member events being given to the Cluster
Manager FSM during a shut down, instread relying on SelfExiting.
This also hardens the test by not downing the node that the current
sharding coordinator is running on as well as fixing a bug in the
probes.
Refs #24113