[ upstream commit c8389d5 ]

[ backporter's notes: hit minor conflicts due to different surrounding
  contexts, solved accepting the combination of changes, and with
  trivial manual adaptations. ]

Let's mimic the same logic already provided by the clustermesh subsystem
part of the Cilium agent, which allows to retrieve key information about
the connection to and data retrieval from each remote cluster. A subsequent
commit is going to wire it to the /clusters API, so that it can then be
accessed through a dedicated CLI.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit f7bd2b4 ]

Wire the API server logic to expose the kvstoremesh API, and register
the handler which returns the remote clusters status information. By
default, the API is served on http://localhost:9889, although the
address can be tuned through a dedicated parameter.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit e670ca6 ]

Extract this logic into a separate function, so that it can be reused
for the kvstoremesh-dbg command as well. Similarly, let's also slightly
refactor and export the NumReadyClusters helper function.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 7cf1f29 ]

Extend the remote clusters output logic to support an additional
verbosity level, to be leveraged by the kvstoremesh-dbg command.

Specifically, supported verbosity levels are:
* verbose: outputs the full information for all clusters;
* brief: outputs the full information for non-ready clusters, and
  a brief one-line summary for ready ones;
* non-ready-only: outputs the full information for non-ready clusters,
  and omits the ready ones.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 4857bc7 ]

Extend the clustermesh-apiserver binary with a new kvstoremesh-dbg
subcommand to interact with the kvstoremesh API, and specifically
allow to query and output the status of the connection to remote
clusters. The command can be invoked through something along the
lines of:

$ kubectl exec -it -n kube-system deploy/clustermesh-apiserver -c kvstoremesh \
    -- clustermesh-apiserver kvstoremesh-dbg status

And outputs the status using the same format of the clustermesh section
reported by cilium-dbg status --all-clusters. By default, the output
includes a brief one-line report for ready clusters, and full information
for non ready ones. Full information for all clusters can be retrieved
specifying the --verbose flag.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit d0af3d7 ]

We shouldn't import testing code into production code, as it can lead to
unexpected side effects due to e.g., init functions. Let's address this
by hard-coding the "PolicyEnforcement" constant, rather than importing
it. This is consistent with the same usage as part of the "config" command.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit cfb3b8a ]

It is intended to be used by CLI tools to retrieve the configuration
files of all remote clusters in a given directory, to be used, e.g.,
for troubleshooting purposes.

While being there, let's also replace the path package with the filepath
one, which is more appropriate in this context, and it would allow to
theoretically handle Windows paths as well.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 2d07cfc ]

[ backporter's notes: replaced cmp.Or usage, as not yet available
  in go 1.21. ]

Troubleshooting etcd connectivity issues, regardless of whether to the
Cilium kvstore or to a remote cluster, is a complex activity, as issues
can concern network connectivity, TLS certificates mismatch, authn/authz
policies and so on.

As an effort to simplify this process, let's introduce a new utility
responsible for performing a set of sanity checks, and outputting the
result in a user-friendly way. This utility is intended to be then
leveraged by dedicated CLI commands integrated with the various
components. More in detail, this utility performs the following
operations:

* Asserts that the etcd configuration can be correctly parsed;
* For each endpoint:
  - Outputs the DNS resolution;
  - Assert that the endpoint is reachable at the network level (i.e.,
    that a TCP connection can be successfully established);
  - When https is enabled, asserts that a TLS connection can be correctly
    established to the endpoint (i.e., that the provided certificates
    are valid); the check includes both server and client (if enabled)
    authentication; additionally outputs TLS specific information;
  - Outputs the version of the endpoint, as returned by GET /version;
* Outputs information regarding Root CAs and client certificates, if
  configured; additionally checks whether the client certificate is
  valid according to the root CAs;
* Asserts that the etcd client can correctly establish a connection;
* Asserts that the heartbeat key can be retrieved, as a basic
  authorization check.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 9654576 ]

Introduce two new cilium-dbg commands, namely "troubleshoot kvstore" and
"troubleshoot clustermesh", responsible for running a set of sanity
checks to help troubleshoot etcd connectivity issues, covering network
connectivity, TLS authentication, authn/authz policies and so on.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 9156e23 ]

As useful to troubleshoot kvstore and clustermesh issues.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 6fae9e7 ]

Extend the clustermesh-apiserver binary with a new clustermesh-dbg
troubleshoot subcommand, responsible for running a set of sanity
checks to help troubleshoot etcd connectivity issues, covering network
connectivity, TLS authentication, authn/authz policies and so on.

The command can be invoked through something along the lines of:

$ kubectl exec -it -n kube-system deploy/clustermesh-apiserver -c apiserver \
  -- clustermesh-apiserver clustermesh-dbg troubleshoot

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit f575f94 ]

Introduce a new troubleshoot subcommand to "clustermesh-apiserver
kvstoremesh-dbg", responsible for running a set of sanity checks
to help troubleshoot etcd connectivity issues, covering network
connectivity, TLS authentication, authn/authz policies and so on.

The command can be invoked through something along the lines of:

$ kubectl exec -it -n kube-system deploy/clustermesh-apiserver -c kvstoremesh \
  -- clustermesh-apiserver kvstoremesh-dbg troubleshoot [--include-local]

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 4172c62 ]

Document the usage of the newly introduced troubleshoot command to
investigate connectivity issues towards the clustermesh control plane
(i.e., etcd) in remote clusters.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 48b36f5 ]

When KVStoreMesh is enabled, this component is responsible for
connecting to the remote clusters. Document the command which
can be used to inspect its status and validate whether connection
are established correctly.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 189e8ba ]

Add a clarification note that the manual steps presented in the guide
are mostly alternative to using the automatic tools described in the
previous section. Additionally, drop the example errors from the TLS
certificates step, as potentially misleading. Users shall leverage
the troubleshoot command instead. Finally, let's fix a couple of typos.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>

[ upstream commit 913e41b ]

They apply only when Cilium is configured in kvstore mode, which is
seldom the case these days. The lack of local information is also not
clustermesh specific, and would imply other serious issues. Moreover,
the given checks would not work, and lead to additional confusion when
Cilium operates in CRD mode. Hence, let's just replace them with the
suggestion of checking whether both Cilium agents and KVStoreMesh
(if enabled) are correctly connected to all remote clusters, and the
synchronization has completed.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>