Before #171 and the csapi package was added, I was able to run Complement tests and see the log output as it runs. But nowadays, all of the test log output is buffered and only printed when all of the tests finish.

This is caused by Go deciding to not stream log output when there are multiple packages present (we have tests and csapi in the ./tests/... directory):

• cmd/go: go test -v does not stream logs when testing multiple packages golang/go#46959
• testing: testing a list of packages blocks the real-time output golang/go#27826

Related:

• Related https://dave.cheney.net/2020/03/10/go-test-v-streaming-output
• Go added the streaming behavior in https://go-review.googlesource.com/c/go/+/127120

I liked the streaming behavior before to see the test progress but it's also important when I try to hook up Element to the homeserver and rooms from Complement to be able to see the log output of room ID's, etc to join and inspect, https://github.com/matrix-org/complement/blob/master/ONBOARDING.md#how-do-i-hook-up-a-matrix-client-like-element-to-the-homeservers-spun-up-by-complement-after-a-test-runs

As a workaround, I can get the log streaming behavior I want by being very selective about the files to run: go test -v -run TestImportHistoricalMessages ./tests/main_test.go ./tests/msc2716_test.go but this is manual process adjusting our generic Synapse test runner every time.

Potential solutions

1. Move all tests to the tests package.
2. Just wait for upstream Go changes although there is a possibility this won't change and just the documentation will be updated with this caveat
3. Run each test package in their own process to still get parallel but still get stream output
4. Can we run tests in parallel without separate packages? Move CSAPI only tests to tests/csapi #171