We already have extensive tests for actual behavior. These tests are written such that we directly call into the event processing code. This is done so all code is executed synchronously which makes them very stable.

A flaw of calling directly into the event processing code is that we don't test the actual lifecycle of the LB-IPAM controller and bugs that might arise in the path between the API client and the event processing logic.

This commit adds a test to exercise the lifecycle of the LB-IPAM controller. To do so we create a hive with smallest setup which still works. LB-IPAM now increments a few atomic counters (only during testing) which we use to glean information about the lifecycle of the controller and if its processing events or not. We start the hive as we would in production. Since everything now runs in its own goroutine we do not have strong guarantees about when the controller will process events. To compensate, relatively large timeouts are used when waiting for something to happen, in an attempt to prevent flaky tests. However due to the nature of the tests, it might still happen, and we will have to keep an eye on it.

When stress testing it locally it seems stable:

stress -o /tmp/test- -p 100 ./lbipam.test -test.run "TestLBIPAMStartupRestartShutdown" -test.timeout=30s
5s: 0 runs so far, 0 failures, 100 active
10s: 0 runs so far, 0 failures, 100 active
15s: 100 runs so far, 0 failures, 100 active
20s: 100 runs so far, 0 failures, 100 active
25s: 103 runs so far, 0 failures, 100 active
30s: 200 runs so far, 0 failures, 100 active
35s: 200 runs so far, 0 failures, 100 active
40s: 300 runs so far, 0 failures, 100 active
45s: 300 runs so far, 0 failures, 100 active
50s: 349 runs so far, 0 failures, 100 active
55s: 400 runs so far, 0 failures, 100 active
1m0s: 400 runs so far, 0 failures, 100 active
1m5s: 500 runs so far, 0 failures, 100 active
1m10s: 500 runs so far, 0 failures, 100 active
1m15s: 586 runs so far, 0 failures, 100 active
1m20s: 600 runs so far, 0 failures, 100 active
1m25s: 600 runs so far, 0 failures, 100 active
1m30s: 700 runs so far, 0 failures, 100 active
1m35s: 700 runs so far, 0 failures, 100 active
1m40s: 800 runs so far, 0 failures, 100 active
1m45s: 800 runs so far, 0 failures, 100 active
1m50s: 800 runs so far, 0 failures, 100 active
1m55s: 900 runs so far, 0 failures, 100 active
2m0s: 900 runs so far, 0 failures, 100 active

This should hopefully catch regressions of #37965 or similar cases.