Testing

To vet In-place Agones Upgrades, we need a combination of unit testing (which are assumed in all PRs), and end-to-end (e2e) testing. The problem with our current e2es is that they assume a particular configuration and test against it. That's good for what they're testing, but we need a different style of test to test upgrades.

From past experience, the best way I have seen to test upgrades is to be doing something and upgrade the system in-place. I propose a system where we keep a cluster under fairly active load, and mutate configuration continuously. As a starting point, imagine three subsystems:

• Wanderer: Every ${PERIOD}, changes configurations randomly within a defined space, e.g. upgrade/downgrade within upgrade horizon, change feature flags, etc. I'm thinking every half hour, giving us a reasonable soak period in between as well, which may help with things like soak testing as well (at least for detecting bigger leaks).

• Producer: Generates load continuously - scale up/down Fleets at random, interact with GameServers via allocation, etc. Some way to ensure there's load on the system. The Producer should keep a continuous metric for operations it does (e.g. allocations) that we can monitor SLO, so if an upgrade causes dips, we notice.

• Monitor: Either configured as alerts, or as an active process monitoring the state of the system, we need to verify that e.g. Agones is healthy, Fleets are healthy, etc.

If we do this right, we could even set this up in a couple of different modes - one with a fast wanderer (e.g. 30 minutes) to make sure we cover the most possible configuration space), and one with a slow wanderer (e.g. a day, a week) to soak test more. The Producer/Monitor for each will look rather similar, just the rate of change will be different.