What happened?

This is a continuation of this discussion: #4260 (comment)

Steps to reproduce

We can probably reproduce it by creating a unit test (which will run with -race) that does the following:

• updates the certificates a couple of times
• meanwhile, makes a couple of client/server calls with TLS using the same tls config

Expected behavior

We expect that changes to certificates in a server take effect on future client connections. Instead, the test described above should detect a race condition as we try to update the same certPools that are already being used by the server (and therefore will be read on new client connections).

Relevant log output

No response

Screenshot

No response

Additional context

The rough proposal would be as follows:

• make the certWatcher the owner of certPools internally, accessible via getter (instead of passing them from options.go)
• store those pools as atomic.Value
• when reloading the certs and adding the to the pools, create new pools via copy first, and store them in atomic.Value
• and most importantly, define tls.Config.GetConfigForClient function by returning a new config with certPools retrieved from certWatcher (similar to how we already define cfg.Get{Client}Certificate functions). This way the server will be using immutable tls configs and cert pools

NB: this probably would not address the issue of reloading root CA for the server or the client, perhaps we should not be even supporting it. GetConfigForClient would only allow supporting reloading of ClientCA on the server, per @tsaarni 's comment.

Jaeger backend version

No response

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

No response

Deployment configs

No response