This repository was archived by the owner on Feb 18, 2025. It is now read-only.
This repository was archived by the owner on Feb 18, 2025. It is now read-only.
Strange martini logs #696
Description
Built up an orchestrator raft cluster with 134(leader)/135/136, then add mysql servers, open web page (135), it takes too long to finish the request, after check the log, I found these error logs
2018/11/07 16:37:39 [DEBUG] raft: Node 10.100.97.135:10008 updated peer set (2): [10.100.97.134:10008 10.100.97.135:10008 10.100.97.136:10008]
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124:56522
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135
.
.
.
.
.
.
.
2018-11-07 16:38:29 ERROR Heath test is failing for over 5 seconds. raft yielding
2018/11/07 16:38:30 [DEBUG] raft-net: 10.100.97.135:10008 accepted connection from: 10.100.97.134:10662
2018-11-07 16:38:30 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:31 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:32 ERROR Heath test is failing for over 5 seconds. raft yielding
2018/11/07 16:38:34 [ERR] raft-net: Failed to flush response: write tcp 10.100.97.135:10008->10.100.97.134:10642: write: broken pipe
2018-11-07 16:38:34 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:40 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:40 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:42 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:42 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:43 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:44 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:38:52 ERROR Heath test is failing for over 5 seconds. raft yielding
2018/11/07 16:38:56 [WARN] raft: Heartbeat timeout from "" reached, but leadership suspended. Will not enter Candidate mode
2018/11/07 16:41:19 [ERR] raft: Failed to make RequestVote RPC to 10.100.97.134:10008: dial tcp 10.100.97.134:10008: i/o timeout
2018/11/07 16:41:19 [ERR] raft: Failed to make RequestVote RPC to 10.100.97.136:10008: dial tcp 10.100.97.136:10008: i/o timeout
2018-11-07 16:41:19 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:41:19 ERROR Heath test is failing for over 5 seconds. raft yielding
2018/11/07 16:41:21 [DEBUG] raft-net: 10.100.97.135:10008 accepted connection from: 10.100.97.134:10716
2018-11-07 16:41:22 ERROR Heath test is failing for over 5 seconds. raft yielding
2018/11/07 16:41:23 http: proxy error: context canceled
2018/11/07 16:41:24 [DEBUG] raft: Votes needed: 2
2018/11/07 16:41:24 [DEBUG] raft: Vote granted from 10.100.97.135:10008. Tally: 1
2018-11-07 16:41:24 ERROR Heath test is failing for over 5 seconds. raft yielding
2018-11-07 16:41:25 FATAL Node is unable to register health. Please check database connnectivity.
Notice: machine cs11 is not 124, the error log is very strange.