-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Closed
Labels
2 - pick nextUsed to mark issues which are next up in the queue to be worked on. The tag is non-bindingUsed to mark issues which are next up in the queue to be worked on. The tag is non-bindingfailedTickets that indicate a CI failure, likely a flakey testTickets that indicate a CI failure, likely a flakey testt:cluster:dc
Milestone
Description
This is a strange failure on the wip-multi-dc-dev
branch. Only seen once in a repeat job.
[info] * akka.cluster.sharding.MultiDcClusterSharding
[JVM-1] 0.538: [GC (Allocation Failure) [PSYoungGen: 16384K->2533K(18944K)] 16384K->4472K(62976K), 0.0140971 secs] [Times: user=0.04 sys=0.02, real=0.01 secs]
[JVM-3] 0.531: [GC (Allocation Failure) [PSYoungGen: 16384K->2559K(18944K)] 16384K->4620K(62976K), 0.0098959 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
[JVM-4] 0.547: [GC (Allocation Failure) [PSYoungGen: 16384K->2529K(18944K)] 16384K->4528K(62976K), 0.0107078 secs] [Times: user=0.01 sys=0.02, real=0.01 secs]
[JVM-2] 0.532: [GC (Allocation Failure) [PSYoungGen: 16384K->2553K(18944K)] 16384K->4594K(62976K), 0.0274696 secs] [Times: user=0.11 sys=0.04, real=0.03 secs]
[JVM-1] 0.903: [GC (Allocation Failure) [PSYoungGen: 18917K->2534K(35328K)] 20856K->5318K(79360K), 0.0111818 secs] [Times: user=0.04 sys=0.04, real=0.01 secs]
[JVM-3] 0.916: [GC (Allocation Failure) [PSYoungGen: 18943K->2541K(35328K)] 21004K->5488K(79360K), 0.0107200 secs] [Times: user=0.02 sys=0.03, real=0.02 secs]
[JVM-4] 0.933: [GC (Allocation Failure) [PSYoungGen: 18913K->2559K(35328K)] 20912K->5367K(79360K), 0.0088949 secs] [Times: user=0.02 sys=0.00, real=0.00 secs]
[JVM-2] 0.954: [GC (Allocation Failure) [PSYoungGen: 18937K->2557K(35328K)] 20978K->5450K(79360K), 0.0086914 secs] [Times: user=0.01 sys=0.02, real=0.01 secs]
[JVM-1] [INFO] [08/30/2017 22:34:20.102] [ScalaTest-main] [akka.remote.Remoting] Starting remoting
[JVM-3] [INFO] [08/30/2017 22:34:20.138] [ScalaTest-main] [akka.remote.Remoting] Starting remoting
[JVM-4] [INFO] [08/30/2017 22:34:20.150] [ScalaTest-main] [akka.remote.Remoting] Starting remoting
[JVM-2] [INFO] [08/30/2017 22:34:20.193] [ScalaTest-main] [akka.remote.Remoting] Starting remoting
[JVM-1] [INFO] [08/30/2017 22:34:20.296] [ScalaTest-main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850]
[JVM-1] 1.790: [GC (Allocation Failure) [PSYoungGen: 35302K->2528K(35328K)] 38086K->12941K(79360K), 0.0209370 secs] [Times: user=0.08 sys=0.00, real=0.02 secs]
[JVM-3] [INFO] [08/30/2017 22:34:20.331] [ScalaTest-main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451]
[JVM-1] [INFO] [08/30/2017 22:34:20.337] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850] dc [DC1] - Starting up...
[JVM-4] [INFO] [08/30/2017 22:34:20.343] [ScalaTest-main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702]
[JVM-4] [INFO] [08/30/2017 22:34:20.362] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702] dc [DC2] - Starting up...
[JVM-3] 1.803: [GC (Allocation Failure) [PSYoungGen: 35309K->2528K(35328K)] 38256K->13169K(79360K), 0.0245666 secs] [Times: user=0.13 sys=0.05, real=0.03 secs]
[JVM-3] [INFO] [08/30/2017 22:34:20.376] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451] dc [DC2] - Starting up...
[JVM-4] 1.831: [GC (Allocation Failure) [PSYoungGen: 35327K->2555K(35328K)] 38135K->13001K(79360K), 0.0197254 secs] [Times: user=0.02 sys=0.08, real=0.02 secs]
[JVM-2] [INFO] [08/30/2017 22:34:20.388] [ScalaTest-main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715]
[JVM-2] 1.853: [GC (Allocation Failure) [PSYoungGen: 35325K->2544K(35328K)] 38218K->13118K(79360K), 0.0161975 secs] [Times: user=0.02 sys=0.07, real=0.02 secs]
[JVM-2] [INFO] [08/30/2017 22:34:20.425] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715] dc [DC1] - Starting up...
[JVM-1] [INFO] [08/30/2017 22:34:20.463] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850] dc [DC1] - Registered cluster JMX MBean [akka:type=Cluster]
[JVM-1] [INFO] [08/30/2017 22:34:20.463] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850] dc [DC1] - Started up successfully
[JVM-1] [INFO] [08/30/2017 22:34:20.499] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850] dc [DC1] - No seed-nodes configured, manual cluster join required
[JVM-1] [WARN] [08/30/2017 22:34:20.502] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[JVM-4] [INFO] [08/30/2017 22:34:20.517] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702] dc [DC2] - Registered cluster JMX MBean [akka:type=Cluster]
[JVM-4] [INFO] [08/30/2017 22:34:20.517] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702] dc [DC2] - Started up successfully
[JVM-3] [INFO] [08/30/2017 22:34:20.518] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451] dc [DC2] - Registered cluster JMX MBean [akka:type=Cluster]
[JVM-3] [INFO] [08/30/2017 22:34:20.518] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451] dc [DC2] - Started up successfully
[JVM-4] [INFO] [08/30/2017 22:34:20.554] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702] dc [DC2] - No seed-nodes configured, manual cluster join required
[JVM-2] [INFO] [08/30/2017 22:34:20.557] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715] dc [DC1] - Registered cluster JMX MBean [akka:type=Cluster]
[JVM-2] [INFO] [08/30/2017 22:34:20.557] [ScalaTest-main] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715] dc [DC1] - Started up successfully
[JVM-4] [WARN] [08/30/2017 22:34:20.557] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[JVM-3] [INFO] [08/30/2017 22:34:20.560] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451] dc [DC2] - No seed-nodes configured, manual cluster join required
[JVM-3] [WARN] [08/30/2017 22:34:20.563] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-4] [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[JVM-2] [INFO] [08/30/2017 22:34:20.595] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715] dc [DC1] - No seed-nodes configured, manual cluster join required
[JVM-2] [WARN] [08/30/2017 22:34:20.598] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-4] [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[JVM-1] [INFO] [08/30/2017 22:34:20.708] [ScalaTest-main] [MultiDcClusterShardingMultiJvmNode1(akka://MultiDcClusterShardingSpec)] Role [first] started with address [akka.tcp://MultiDcClusterShardingSpec@a4.moxie:45850]
[JVM-2] [INFO] [08/30/2017 22:34:20.773] [ScalaTest-main] [MultiDcClusterShardingMultiJvmNode2(akka://MultiDcClusterShardingSpec)] Role [second] started with address [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:41715]
[JVM-3] [INFO] [08/30/2017 22:34:20.776] [ScalaTest-main] [MultiDcClusterShardingMultiJvmNode3(akka://MultiDcClusterShardingSpec)] Role [third] started with address [akka.tcp://MultiDcClusterShardingSpec@a2.moxie:59451]
[JVM-4] [INFO] [08/30/2017 22:34:20.781] [ScalaTest-main] [MultiDcClusterShardingMultiJvmNode4(akka://MultiDcClusterShardingSpec)] Role [fourth] started with address [akka.tcp://MultiDcClusterShardingSpec@a5.moxie:59702]
[first] Run starting. Expected test count is: 5
[first] MultiDcClusterShardingMultiJvmNode1:
[first] Cluster sharding in multi data center cluster
[first] [INFO] [08/30/2017 22:34:20.927] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Node [akka.tcp://MultiDcClusterShardingSpec@first] is JOINING, roles [dc-DC1]
[second] Run starting. Expected test count is: 5
[second] MultiDcClusterShardingMultiJvmNode2:
[third] Run starting. Expected test count is: 5
[first] [INFO] [08/30/2017 22:34:20.942] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Leader is moving node [akka.tcp://MultiDcClusterShardingSpec@first] to [Up]
[third] MultiDcClusterShardingMultiJvmNode3:
[fourth] Run starting. Expected test count is: 5
[fourth] MultiDcClusterShardingMultiJvmNode4:
[second] Cluster sharding in multi data center cluster
[third] Cluster sharding in multi data center cluster
[fourth] Cluster sharding in multi data center cluster
[first] 2.450: [GC (GCLocker Initiated GC) [PSYoungGen: 35296K->2528K(40960K)] 45709K->20928K(84992K), 0.0292981 secs] [Times: user=0.16 sys=0.04, real=0.03 secs]
[first] [WARN] [08/30/2017 22:34:21.080] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-16] [akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton/coordinator] received dead letter from Actor[akka://MultiDcClusterShardingSpec/system/sharding/Entity#-1153603649]: Register(Actor[akka://MultiDcClusterShardingSpec/system/sharding/Entity#-1153603649])
[first] [INFO] [08/30/2017 22:34:21.079] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton/coordinator] Message [akka.cluster.sharding.ShardCoordinator$Internal$Register] from Actor[akka://MultiDcClusterShardingSpec/system/sharding/Entity#-1153603649] to Actor[akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton/coordinator] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[first] [INFO] [08/30/2017 22:34:21.088] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-21] [akka.tcp://MultiDcClusterShardingSpec@first/system/sharding/EntityCoordinator] Singleton manager starting singleton actor [akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton]
[first] [INFO] [08/30/2017 22:34:21.090] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-21] [akka.tcp://MultiDcClusterShardingSpec@first/system/sharding/EntityCoordinator] ClusterSingletonManager state change [Start -> Oldest]
[first] [INFO] [08/30/2017 22:34:21.134] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-21] [akka.tcp://MultiDcClusterShardingSpec@first/system/sharding/EntityCoordinator/singleton/coordinator] Sharding Coordinator was moved to the active state State(Map(),Map(),Set(),Set(),false)
[second] 2.694: [GC (Allocation Failure) [PSYoungGen: 35312K->2528K(40960K)] 45886K->21245K(84992K), 0.0173782 secs] [Times: user=0.08 sys=0.10, real=0.02 secs]
[first] [INFO] [08/30/2017 22:34:21.389] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Node [akka.tcp://MultiDcClusterShardingSpec@second] is JOINING, roles [dc-DC1]
[first] [INFO] [08/30/2017 22:34:21.508] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Leader is moving node [akka.tcp://MultiDcClusterShardingSpec@second] to [Up]
[second] [INFO] [08/30/2017 22:34:21.521] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@second] dc [DC1] - Welcome from [akka.tcp://MultiDcClusterShardingSpec@first]
[second] [INFO] [08/30/2017 22:34:21.636] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.tcp://MultiDcClusterShardingSpec@second/system/sharding/EntityCoordinator] ClusterSingletonManager state change [Start -> Younger]
[third] 3.223: [GC (Allocation Failure) [PSYoungGen: 35296K->2544K(40960K)] 45937K->21342K(84992K), 0.0292573 secs] [Times: user=0.11 sys=0.08, real=0.03 secs]
[first] [INFO] [08/30/2017 22:34:21.873] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Node [akka.tcp://MultiDcClusterShardingSpec@third] is JOINING, roles [dc-DC2]
[third] [INFO] [08/30/2017 22:34:21.940] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Welcome from [akka.tcp://MultiDcClusterShardingSpec@first]
[third] [INFO] [08/30/2017 22:34:22.577] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Leader is moving node [akka.tcp://MultiDcClusterShardingSpec@third] to [Up]
[third] [INFO] [08/30/2017 22:34:22.590] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-19] [akka.tcp://MultiDcClusterShardingSpec@third/system/cluster/core/daemon/crossDcHeartbeatSender] Cross DC heartbeat becoming ACTIVE on this node (for DC: DC2), monitoring other DCs oldest nodes
[third] [INFO] [08/30/2017 22:34:22.600] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.tcp://MultiDcClusterShardingSpec@third/system/sharding/EntityCoordinator] Singleton manager starting singleton actor [akka://MultiDcClusterShardingSpec/system/sharding/EntityCoordinator/singleton]
[third] [INFO] [08/30/2017 22:34:22.602] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.tcp://MultiDcClusterShardingSpec@third/system/sharding/EntityCoordinator] ClusterSingletonManager state change [Start -> Oldest]
[third] [INFO] [08/30/2017 22:34:22.645] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.tcp://MultiDcClusterShardingSpec@third/system/sharding/EntityCoordinator/singleton/coordinator] Sharding Coordinator was moved to the active state State(Map(),Map(),Set(),Set(),false)
[fourth] 4.183: [GC (GCLocker Initiated GC) [PSYoungGen: 35323K->2555K(40960K)] 45770K->21089K(84992K), 0.0284755 secs] [Times: user=0.13 sys=0.05, real=0.03 secs]
[first] [INFO] [08/30/2017 22:34:22.826] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@first] dc [DC1] - Node [akka.tcp://MultiDcClusterShardingSpec@fourth] is JOINING, roles [dc-DC2]
[fourth] [INFO] [08/30/2017 22:34:22.904] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@fourth] dc [DC2] - Welcome from [akka.tcp://MultiDcClusterShardingSpec@first]
[third] [INFO] [08/30/2017 22:34:23.600] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:24.571] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:25.570] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-19] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:26.570] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-20] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:27.569] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-19] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:28.572] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:29.571] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:30.568] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-2] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:31.570] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-19] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:32.570] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:33.569] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[third] [INFO] [08/30/2017 22:34:34.569] [MultiDcClusterShardingSpec-akka.actor.default-dispatcher-17] [akka.cluster.Cluster(akka://MultiDcClusterShardingSpec)] Cluster Node [akka.tcp://MultiDcClusterShardingSpec@third] dc [DC2] - Ignoring received gossip from unreachable [UniqueAddress(akka.tcp://MultiDcClusterShardingSpec@fourth,-1522030475)]
[fourth] - must join cluster (on node 'fourth', class akka.cluster.sharding.MultiDcClusterShardingMultiJvmNode4) *** FAILED ***
[fourth] false was not true (MultiDcClusterShardingSpec.scala:92)
fourth
is not moved to Up in the joining phase. The "Ignoring received gossip from unreachable" is probably because fourth is not a known member in third
yet. Not because real unreachable since there is no "Marking node(s) as UNREACHABLE" log entry.
It joins to first
, so why doesn't first gossip that to third
?
I'll enable debug logging for this test.
Metadata
Metadata
Assignees
Labels
2 - pick nextUsed to mark issues which are next up in the queue to be worked on. The tag is non-bindingUsed to mark issues which are next up in the queue to be worked on. The tag is non-bindingfailedTickets that indicate a CI failure, likely a flakey testTickets that indicate a CI failure, likely a flakey testt:cluster:dc