Hello

Neighborhood check is expensive when running on the scale. For instance:

• neighbor discovery creates a load on the API server proportional to the cluster size and frequency of the checks. The same applies to node watchers
• we have n^2 TCP/IP handshakes every 5s, where n is the number of nodes. This also creates more packets into node network queues (FIFO queues), and regular network traffic for production applications could be hit by a latency increase

My proposal would be:

1. configurable check scheduling, so we can control to reduce the check frequency to once per minute, for example, when needed
2. optimize the way we query the API server for discovery information; one option is a SWIM-based solution like hashicorp/memberlist

While the first point is straightforward and does not change the behavior much, the latter requires discussion