Skip to content

failed: MultiDcClusterSharding #23582

@patriknw

Description

@patriknw

This is a strange failure on the wip-multi-dc-dev branch. Only seen once in a repeat job.

[info] * akka.cluster.sharding.MultiDcClusterSharding
[JVM-1] 0.538: [GC (Allocation Failure) [PSYoungGen: 16384K->2533K(18944K)] 16384K->4472K(62976K), 0.0140971 secs] [Times: user=0.04 sys=0.02, real=0.01 secs] 
[JVM-3] 0.531: [GC (Allocation Failure) [PSYoungGen: 16384K->2559K(18944K)] 16384K->4620K(62976K), 0.0098959 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] 
[JVM-4] 0.547: [GC (Allocation Failure) [PSYoungGen: 16384K->2529K(18944K)] 16384K->4528K(62976K), 0.0107078 secs] [Times: user=0.01 sys=0.02, real=0.01 secs] 
[JVM-2] 0.532: [GC (Allocation Failure) [PSYoungGen: 16384K->2553K(18944K)] 16384K->4594K(62976K), 0.0274696 secs] [Times: user=0.11 sys=0.04, real=0.03 secs] 
[JVM-1] 0.903: [GC (Allocation Failure) [PSYoungGen: 18917K->2534K(35328K)] 20856K->5318K(79360K), 0.0111818 secs] [Times: user=0.04 sys=0.04, real=0.01 secs] 
[JVM-3] 0.916: [GC (Allocation Failure) [PSYoungGen: 18943K->2541K(35328K)] 21004K->5488K(79360K), 0.0107200 secs] [Times: user=0.02 sys=0.03, real=0.02 secs] 
[JVM-4] 0.933: [GC (Allocation Failure) [PSYoungGen: 18913K->2559K(35328K)] 20912K->5367K(79360K), 0.0088949 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
[JVM-2] 0.954: [GC (Allocation Failure) [PSYoungGen: 18937K->2557K(35328K)] 20978K->5450K(79360K), 0.0086914 secs] [Times: user=0.01 sys=0.02, real=0.01 secs] 
[JVM-1] [INFO] [08/30/2017 22:34:20.102] [ScalaTest-main] [akka.remote.Remoting] Starting remoting
[JVM-3] [INFO] [08/30/2017 22:34:20.138] [ScalaTest-main] [akka.remote.Remoting] Starting remoting
[JVM-4] [INFO] [08/30/2017 22:34:20.150] [ScalaTest-main] [akka.remote.Remoting] Starting remoting
[JVM-2] [INFO] [08/30/2017 22:34:20.193] [ScalaTest-main] [akka.remote.Remoting] Starting remoting
[JVM-1] [INFO] [08/30/2017 22:34:20.296] [ScalaTest-main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850]
[JVM-1] 1.790: [GC (Allocation Failure) [PSYoungGen: 35302K->2528K(35328K)] 38086K->12941K(79360K), 0.0209370 secs] [Times: user=0.08 sys=0.00, real=0.02 secs] 
[JVM-3] [INFO] [08/30/2017 22:34:20.331] [ScalaTest-main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451]
[JVM-1] [INFO] [08/30/2017 22:34:20.337] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850] dc [DC1] - Starting up...
[JVM-4] [INFO] [08/30/2017 22:34:20.343] [ScalaTest-main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702]
[JVM-4] [INFO] [08/30/2017 22:34:20.362] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702] dc [DC2] - Starting up...
[JVM-3] 1.803: [GC (Allocation Failure) [PSYoungGen: 35309K->2528K(35328K)] 38256K->13169K(79360K), 0.0245666 secs] [Times: user=0.13 sys=0.05, real=0.03 secs] 
[JVM-3] [INFO] [08/30/2017 22:34:20.376] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451] dc [DC2] - Starting up...
[JVM-4] 1.831: [GC (Allocation Failure) [PSYoungGen: 35327K->2555K(35328K)] 38135K->13001K(79360K), 0.0197254 secs] [Times: user=0.02 sys=0.08, real=0.02 secs] 
[JVM-2] [INFO] [08/30/2017 22:34:20.388] [ScalaTest-main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715]
[JVM-2] 1.853: [GC (Allocation Failure) [PSYoungGen: 35325K->2544K(35328K)] 38218K->13118K(79360K), 0.0161975 secs] [Times: user=0.02 sys=0.07, real=0.02 secs] 
[JVM-2] [INFO] [08/30/2017 22:34:20.425] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715] dc [DC1] - Starting up...
[JVM-1] [INFO] [08/30/2017 22:34:20.463] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850] dc [DC1] - Registered cluster JMX MBean [akka:type=Cluster]
[JVM-1] [INFO] [08/30/2017 22:34:20.463] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850] dc [DC1] - Started up successfully
[JVM-1] [INFO] [08/30/2017 22:34:20.499] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850] dc [DC1] - No seed-nodes configured, manual cluster join required
[JVM-1] [WARN] [08/30/2017 22:34:20.502] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[JVM-4] [INFO] [08/30/2017 22:34:20.517] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702] dc [DC2] - Registered cluster JMX MBean [akka:type=Cluster]
[JVM-4] [INFO] [08/30/2017 22:34:20.517] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702] dc [DC2] - Started up successfully
[JVM-3] [INFO] [08/30/2017 22:34:20.518] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451] dc [DC2] - Registered cluster JMX MBean [akka:type=Cluster]
[JVM-3] [INFO] [08/30/2017 22:34:20.518] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451] dc [DC2] - Started up successfully
[JVM-4] [INFO] [08/30/2017 22:34:20.554] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702] dc [DC2] - No seed-nodes configured, manual cluster join required
[JVM-2] [INFO] [08/30/2017 22:34:20.557] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715] dc [DC1] - Registered cluster JMX MBean [akka:type=Cluster]
[JVM-2] [INFO] [08/30/2017 22:34:20.557] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715] dc [DC1] - Started up successfully
[JVM-4] [WARN] [08/30/2017 22:34:20.557] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[JVM-3] [INFO] [08/30/2017 22:34:20.560] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451] dc [DC2] - No seed-nodes configured, manual cluster join required
[JVM-3] [WARN] [08/30/2017 22:34:20.563] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-4] [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[JVM-2] [INFO] [08/30/2017 22:34:20.595] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715] dc [DC1] - No seed-nodes configured, manual cluster join required
[JVM-2] [WARN] [08/30/2017 22:34:20.598] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-4] [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[JVM-1] [INFO] [08/30/2017 22:34:20.708] [ScalaTest-main] [MultiDcClusterShardingMultiJvmNode1(akka://MultiDcClusterShardingSpec)] Role [first] started with address [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850]
[JVM-2] [INFO] [08/30/2017 22:34:20.773] [ScalaTest-main] [MultiDcClusterShardingMultiJvmNode2(akka://MultiDcClusterShardingSpec)] Role [second] started with address [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715]
[JVM-3] [INFO] [08/30/2017 22:34:20.776] [ScalaTest-main] [MultiDcClusterShardingMultiJvmNode3(akka://MultiDcClusterShardingSpec)] Role [third] started with address [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451]
[JVM-4] [INFO] [08/30/2017 22:34:20.781] [ScalaTest-main] [MultiDcClusterShardingMultiJvmNode4(akka://MultiDcClusterShardingSpec)] Role [fourth] started with address [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702]
[first] Run starting. Expected test count is: 5
[first] MultiDcClusterShardingMultiJvmNode1:
[first] Cluster sharding in multi data center cluster
[first] [INFO] [08/30/2017 22:34:20.927] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Node [akka.tcp://MultiDcClusterShardingSpec@first] is JOINING, roles [dc-DC1]
[second] Run starting. Expected test count is: 5
[second] MultiDcClusterShardingMultiJvmNode2:
[third] Run starting. Expected test count is: 5
[first] [INFO] [08/30/2017 22:34:20.942] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Leader is moving node [akka.tcp://MultiDcClusterShardingSpec@first] to [Up]
[third] MultiDcClusterShardingMultiJvmNode3:
[fourth] Run starting. Expected test count is: 5
[fourth] MultiDcClusterShardingMultiJvmNode4:
[second] Cluster sharding in multi data center cluster
[third] Cluster sharding in multi data center cluster
[fourth] Cluster sharding in multi data center cluster
[first] 2.450: [GC (GCLocker Initiated GC) [PSYoungGen: 35296K->2528K(40960K)] 45709K->20928K(84992K), 0.0292981 secs] [Times: user=0.16 sys=0.04, real=0.03 secs] 
[first] [WARN] [08/30/2017 22:34:21.080] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-16] [akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton/coordinator] received dead letter from Actor[akka://MultiDcClusterShardingSpec/system/sharding/Entity#-1153603649]: Register(Actor[akka://MultiDcClusterShardingSpec/system/sharding/Entity#-1153603649])
[first] [INFO] [08/30/2017 22:34:21.079] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton/coordinator] Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka://MultiDcClusterShardingSpec/system/sharding/Entity#-1153603649] to Actor[akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton/coordinator] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[first] [INFO] [08/30/2017 22:34:21.088] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-21] [akka.tcp://MultiDcClusterShardingSpec@first/system/sharding/EntityCoordinator] Singleton manager starting singleton actor [akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton]
[first] [INFO] [08/30/2017 22:34:21.090] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-21] [akka.tcp://MultiDcClusterShardingSpec@first/system/sharding/EntityCoordinator] ClusterSingletonManager state change [Start -> Oldest]
[first] [INFO] [08/30/2017 22:34:21.134] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-21] [akka.tcp://MultiDcClusterShardingSpec@first/system/sharding/EntityCoordinator/singleton/coordinator] Sharding Coordinator was moved to the active state State(Map(),Map(),Set(),Set(),false)
[second] 2.694: [GC (Allocation Failure) [PSYoungGen: 35312K->2528K(40960K)] 45886K->21245K(84992K), 0.0173782 secs] [Times: user=0.08 sys=0.10, real=0.02 secs] 
[first] [INFO] [08/30/2017 22:34:21.389] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Node [akka.tcp://MultiDcClusterShardingSpec@second] is JOINING, roles [dc-DC1]
[first] [INFO] [08/30/2017 22:34:21.508] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Leader is moving node [akka.tcp://MultiDcClusterShardingSpec@second] to [Up]
[second] [INFO] [08/30/2017 22:34:21.521] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@second] dc [DC1] - Welcome from [akka.tcp://MultiDcClusterShardingSpec@first]
[second] [INFO] [08/30/2017 22:34:21.636] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.tcp://MultiDcClusterShardingSpec@second/system/sharding/EntityCoordinator] ClusterSingletonManager state change [Start -> Younger]
[third] 3.223: [GC (Allocation Failure) [PSYoungGen: 35296K->2544K(40960K)] 45937K->21342K(84992K), 0.0292573 secs] [Times: user=0.11 sys=0.08, real=0.03 secs] 
[first] [INFO] [08/30/2017 22:34:21.873] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Node [akka.tcp://MultiDcClusterShardingSpec@third] is JOINING, roles [dc-DC2]
[third] [INFO] [08/30/2017 22:34:21.940] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Welcome from [akka.tcp://MultiDcClusterShardingSpec@first]
[third] [INFO] [08/30/2017 22:34:22.577] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Leader is moving node [akka.tcp://MultiDcClusterShardingSpec@third] to [Up]
[third] [INFO] [08/30/2017 22:34:22.590] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-19] [akka.tcp://MultiDcClusterShardingSpec@third/system/cluster/core/daemon/crossDcHeartbeatSender] Cross DC heartbeat becoming ACTIVE on this node (for DC: DC2), monitoring other DCs oldest nodes
[third] [INFO] [08/30/2017 22:34:22.600] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.tcp://MultiDcClusterShardingSpec@third/system/sharding/EntityCoordinator] Singleton manager starting singleton actor [akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton]
[third] [INFO] [08/30/2017 22:34:22.602] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.tcp://MultiDcClusterShardingSpec@third/system/sharding/EntityCoordinator] ClusterSingletonManager state change [Start -> Oldest]
[third] [INFO] [08/30/2017 22:34:22.645] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.tcp://MultiDcClusterShardingSpec@third/system/sharding/EntityCoordinator/singleton/coordinator] Sharding Coordinator was moved to the active state State(Map(),Map(),Set(),Set(),false)
[fourth] 4.183: [GC (GCLocker Initiated GC) [PSYoungGen: 35323K->2555K(40960K)] 45770K->21089K(84992K), 0.0284755 secs] [Times: user=0.13 sys=0.05, real=0.03 secs] 
[first] [INFO] [08/30/2017 22:34:22.826] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Node [akka.tcp://MultiDcClusterShardingSpec@fourth] is JOINING, roles [dc-DC2]
[fourth] [INFO] [08/30/2017 22:34:22.904] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@fourth] dc [DC2] - Welcome from [akka.tcp://MultiDcClusterShardingSpec@first]
[third] [INFO] [08/30/2017 22:34:23.600] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:24.571] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:25.570] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-19] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:26.570] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:27.569] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-19] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:28.572] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:29.571] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:30.568] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:31.570] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-19] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:32.570] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:33.569] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[third] [INFO] [08/30/2017 22:34:34.569] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)] 
[fourth] - must join cluster (on node 'fourth', class akka.cluster.sharding.MultiDcClusterShardingMultiJvmNode4) *** FAILED ***
[fourth]   false was not true (MultiDcClusterShardingSpec.scala:92)

fourth is not moved to Up in the joining phase. The "Ignoring received gossip from unreachable" is probably because fourth is not a known member in third yet. Not because real unreachable since there is no "Marking node(s) as UNREACHABLE" log entry.

It joins to first, so why doesn't first gossip that to third?

I'll enable debug logging for this test.

Metadata

Metadata

Assignees

Labels

2 - pick nextUsed to mark issues which are next up in the queue to be worked on. The tag is non-bindingfailedTickets that indicate a CI failure, likely a flakey testt:cluster:dc

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions