The recent releases of Concourse have been a bit rocky, and I think some of the issues could have been avoided if we wrote tests for a fresh-install of Concourse for our supported installation methods. While BOSH is probably the first method that comes to mind (#2125 and #2029 are both examples of BOSH installation problems identified immediately after release), we should test all our supported methods.

We might not need to ruin this test as a part of our regular develop group, but probably as a pre-publish job / test?