-
Notifications
You must be signed in to change notification settings - Fork 939
Raft Reverse-Proxy doesn't work when UseSSL = true #1343
Description
This issue is similar to #873
Actual behaviour:
When orchestrator.conf.json
is configured that way:
{
"HTTPAdvertise": "http://{{ orchestrator_hostname }}:{{ orchestrator_port }}",
"UseSSL": false
}
Then reverse-proxy works fine and curl - D - http://{orchestrator_ip}:{orchestrator_port}/api/clusters
request returns correct information about known clusters both for a raft leader and its followers.
But when it is configured that way:
{
"HTTPAdvertise": "https://{{ orchestrator_hostname }}:{{ orchestrator_port }}",
"UseSSL": true
}
Then curl -D - -k https://{orchestrator_ip}:{orchestrator_port}/api/clusters
request returns correct information only if it was sent to a raft leader. If it's sent to a follower, it returns 502 Bad Gateway
and Orchestrator logs say: remote error: tls: bad certificate
.
Expected behaviour:
reverse-proxy
should work properly, so any follower should be able to provide requested information from its leader.
Root cause analysis:
orchestrator/go/http/raft_reverse_proxy.go
Lines 46 to 47 in f818d8a
proxy := httputil.NewSingleHostReverseProxy(url) | |
proxy.ServeHTTP(w, r) |
Creates a new proxy without any HTTP transport and starts serving HTTP . Since no transport is configured, a default HTTP transport is created automatically:
func (p *ReverseProxy) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
transport := p.Transport
if transport == nil {
transport = http.DefaultTransport
}
This works when SSL is disabled, but it breaks routing logic in the middle when it is enabled.
Workaround:
Sending all requests directly to a raft leader works pretty well both with and without SSL.
Solution proposal:
A suitable HTTP transport is already being created for raft health requests:
orchestrator/go/raft/http_client.go
Lines 37 to 68 in f818d8a
httpTimeout := time.Duration(config.ActiveNodeExpireSeconds) * time.Second | |
dialTimeout := func(network, addr string) (net.Conn, error) { | |
return net.DialTimeout(network, addr, httpTimeout) | |
} | |
tlsConfig := &tls.Config{ | |
InsecureSkipVerify: config.Config.SSLSkipVerify, | |
} | |
if config.Config.UseSSL { | |
caPool, err := ssl.ReadCAFile(config.Config.SSLCAFile) | |
if err != nil { | |
return err | |
} | |
tlsConfig.RootCAs = caPool | |
if config.Config.UseMutualTLS { | |
var sslPEMPassword []byte | |
if ssl.IsEncryptedPEM(config.Config.SSLPrivateKeyFile) { | |
sslPEMPassword = ssl.GetPEMPassword(config.Config.SSLPrivateKeyFile) | |
} | |
if err := ssl.AppendKeyPairWithPassword(tlsConfig, config.Config.SSLCertFile, config.Config.SSLPrivateKeyFile, sslPEMPassword); err != nil { | |
return err | |
} | |
} | |
} | |
httpTransport := &http.Transport{ | |
TLSClientConfig: tlsConfig, | |
Dial: dialTimeout, | |
ResponseHeaderTimeout: httpTimeout, | |
} | |
httpClient = &http.Client{Transport: httpTransport} |
This code can be reused to get a proper HTTP transport for a reverse-proxy as well.