Description

This PR extracts the initialization logic for the Hubble metrics server and the flow processor into its own Hive cell.
Since this work required abstracting the mechanism for retrieving certificates for the Hubble metrics server using certloader.WatchedServerConfig, I also re-used it for the Hubble server certificates.

Two notable behaviour changes:

• The metrics server no longer blocks Hubble from starting while waiting for TLS certificates to become available.
• The metrics server now reports warnings/errors in cilium status.
• The metrics server and certloader tls cert watcher now reports their job status in the health table.

Additionally, the default value for the --hubble-disable-tls flag is now true. The value in helm options is unchanged. This is to prevent errors in hive tests returned by the certloader dependency requiring a filepath set when enabled at hive startup time. Before we did not face issues because the TLS config watcher was always setup in the launch method of Hubble, after the cell has been started. The helm chart always sets the value explicitly, therefore there is no behaviour change caused by this.

See individual commits for details.

Related: #34501
cc @kaworu

Extract the Hubble metrics server into its own Hive cell
Include more details about the Hubble metrics server state in cilium-status

Richer cilium status reporting

We now record and report warnings and errors when starting/running the metrics server.
Previously, we would always report OK even if the server crashed at some point. Additionally, now that Hubble no longer blocks at initialization waiting for the metrics server TLS certificates to become ready, we can report the state of Hubble and the metrics server independently.

Example:

$ cilium status
...
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 31.79   Metrics: Warning (Waiting for TLS certificates to become available)

We could think of expending this in the future to report more information, such as how many metrics handlers are enabled, and even which ones.

Additionally, by using one-shot jobs to register long-lived processes such as the metrics server and certloader tls watchers, we get entries in the health table:

$ cilium status --all-health
...
├── hubble
│   ├── hubble-metrics
│   │   ├── job-certloader-tls          [OK] Running (8m33s, x1)
│   │   └── job-hubble-metrics-server   [OK] Running (8m33s, x1)
│   ├── job-certloader-tls              [STOPPED] Running (8m33s, x1)
│   └── job-hubble                      [STOPPED] Running (8m33s, x1)

Certloader cell provider

While working on this, I noticed that the process of creating a FutureWatchedServerConfig and waiting for TLS certificates to become ready was duplicated for both the Hubble metrics server and the observer server. I took this opportunity to refactor that logic into a new certloader.NewWatchedServerConfigPromise constructor that can be used with cell.Provide and cell.ProvidePrivate.

At first, I considered adding a new "reusable" cell, similar to how the pprof cell has been refactored lately and allows providing custom flags for it. The difficulty here is in avoiding collisions/replacement with the provided type, here a promise over a WatchedServerConfig, when the cell is being used multiple times in the same hive. By only providing the constructor, its possible for a cell author to wrap the return type and provide that as a dependency. This is how we are doing it here with both the Hubble metrics server and the Hubble server cells.