Skip to content
This repository was archived by the owner on Feb 18, 2025. It is now read-only.
This repository was archived by the owner on Feb 18, 2025. It is now read-only.

Raft Reverse-Proxy doesn't work when UseSSL = true #1343

@akatashev

Description

@akatashev

This issue is similar to #873

Actual behaviour:

When orchestrator.conf.json is configured that way:

{
  "HTTPAdvertise": "http://{{ orchestrator_hostname }}:{{ orchestrator_port }}",
  "UseSSL": false
}

Then reverse-proxy works fine and curl - D - http://{orchestrator_ip}:{orchestrator_port}/api/clusters request returns correct information about known clusters both for a raft leader and its followers.

But when it is configured that way:

{
  "HTTPAdvertise": "https://{{ orchestrator_hostname }}:{{ orchestrator_port }}",
  "UseSSL": true
}

Then curl -D - -k https://{orchestrator_ip}:{orchestrator_port}/api/clusters request returns correct information only if it was sent to a raft leader. If it's sent to a follower, it returns 502 Bad Gateway and Orchestrator logs say: remote error: tls: bad certificate.

Expected behaviour:

reverse-proxy should work properly, so any follower should be able to provide requested information from its leader.

Root cause analysis:

proxy := httputil.NewSingleHostReverseProxy(url)
proxy.ServeHTTP(w, r)

Creates a new proxy without any HTTP transport and starts serving HTTP . Since no transport is configured, a default HTTP transport is created automatically:

   func (p *ReverseProxy) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
   	transport := p.Transport
    	if transport == nil {
    		transport = http.DefaultTransport
    	}

This works when SSL is disabled, but it breaks routing logic in the middle when it is enabled.

Workaround:

Sending all requests directly to a raft leader works pretty well both with and without SSL.

Solution proposal:

A suitable HTTP transport is already being created for raft health requests:

httpTimeout := time.Duration(config.ActiveNodeExpireSeconds) * time.Second
dialTimeout := func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, httpTimeout)
}
tlsConfig := &tls.Config{
InsecureSkipVerify: config.Config.SSLSkipVerify,
}
if config.Config.UseSSL {
caPool, err := ssl.ReadCAFile(config.Config.SSLCAFile)
if err != nil {
return err
}
tlsConfig.RootCAs = caPool
if config.Config.UseMutualTLS {
var sslPEMPassword []byte
if ssl.IsEncryptedPEM(config.Config.SSLPrivateKeyFile) {
sslPEMPassword = ssl.GetPEMPassword(config.Config.SSLPrivateKeyFile)
}
if err := ssl.AppendKeyPairWithPassword(tlsConfig, config.Config.SSLCertFile, config.Config.SSLPrivateKeyFile, sslPEMPassword); err != nil {
return err
}
}
}
httpTransport := &http.Transport{
TLSClientConfig: tlsConfig,
Dial: dialTimeout,
ResponseHeaderTimeout: httpTimeout,
}
httpClient = &http.Client{Transport: httpTransport}

This code can be reused to get a proper HTTP transport for a reverse-proxy as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions