Shard region registration to more potential oldest, #28416 #28470
Conversation
* Keep track of Leaving and Exiting members in ShardRegion and attempt to register to coordinator at several of the oldest if they have status Leaving and Exiting. Include all up to and including the first member with status Up. * Sending to wrong node doesn't matter, will be suppressed deadLetter. * Same for the GracefulShutdownReq which already had that intention by sending to 2 oldest.
| /** | ||
| * Test for issue #28416 | ||
| */ | ||
| object ClusterShardingRegistrationCoordinatedShutdownSpec extends MultiNodeConfig { |
There was a problem hiding this comment.
I wasn't able to reproduce the originally reported scenario, and this test doesn't fail without the changes. I'm out of ideas for how to reproduce the problem. It was support case 13482.
|
Test PASSed. |
| case MemberLeft(m) => | ||
| addMember(m) | ||
| case MemberExited(m) => | ||
| addMember(m) |
There was a problem hiding this comment.
Hmm, so it would go directly from Joining/WeaklyUp to Left or Exited and therefore we need to add it for those as well? Why do we need the memberStatusOfInterest if we only ever add those statuses anyway?
There was a problem hiding this comment.
for MemberLeft and MemberExited they will already be in the set, but we want to replace them since the status changed
There was a problem hiding this comment.
memberStatusOfInterest is also used when receiving the CurrentClusterState. it's true that it wouldn't be necessary to check memberStatusOfInterest in addMember but I thought that would be more future proof
There was a problem hiding this comment.
Ok, thanks for clarification.
| } | ||
| } | ||
|
|
||
| select(Nil, membersByAge).map(m => context.actorSelection(RootActorPath(m.address).toString + coordinatorPath)) |
There was a problem hiding this comment.
Um, so always all nodes that are up with oldest first? Shouldn't it be capped for larger clusters?
There was a problem hiding this comment.
+1, in that case this could take a while.
There was a problem hiding this comment.
always all nodes that are up with oldest first
no, that's not what this does. it collects nodes until it detects one that is Up and the result includes that single Up.
example
(Up, Up, Up) => (Up)
(leaving, Up, Up) => (Leaving, Up)
(leaving, Exiting, Up, Up) => (Leaving, Exiting, Up)
There was a problem hiding this comment.
Ah, I missed the early return and thought the isEmpty was the single termination. Gotcha.
* Keep track of Leaving and Exiting members in ShardRegion and attempt to register to coordinator at several of the oldest if they have status Leaving and Exiting. Include all up to and including the first member with status Up. * Sending to wrong node doesn't matter, will be suppressed deadLetter. * Same for the GracefulShutdownReq which already had that intention by sending to 2 oldest. migrated from akka/akka#28470
* Keep track of Leaving and Exiting members in ShardRegion and attempt to register to coordinator at several of the oldest if they have status Leaving and Exiting. Include all up to and including the first member with status Up. * Sending to wrong node doesn't matter, will be suppressed deadLetter. * Same for the GracefulShutdownReq which already had that intention by sending to 2 oldest. migrated from akka/akka#28470
* Avoid dead letter for rebalance timeout msg migrated from akka/akka#28274 * Reconsider cluster.role.<role-name>.min-nr-of-members fallback migrated from akka/akka#28203 * Use dedicated message for PersistentShardCoodinator termination instead of PoisonPill migrated from akka/akka#28104 * Don't initialize durable keys when rememberEntities not used migrated from akka/akka#28568 * Improve error message when coordinator not found migrated from akka/akka#28576 * Shard region registration to more potential oldest * Keep track of Leaving and Exiting members in ShardRegion and attempt to register to coordinator at several of the oldest if they have status Leaving and Exiting. Include all up to and including the first member with status Up. * Sending to wrong node doesn't matter, will be suppressed deadLetter. * Same for the GracefulShutdownReq which already had that intention by sending to 2 oldest. migrated from akka/akka#28470 * Log non-completed rebalance at warning level migrated from akka/akka#28335 * api approval fix * review fixes Co-authored-by: Aaron Stannard <aaron@petabridge.com>
attempt to register to coordinator at several of the oldest if
they have status Leaving and Exiting. Include all up to and
including the first member with status Up.
sending to 2 oldest.
Refs #28416