-
Notifications
You must be signed in to change notification settings - Fork 8
Description
The CI workflow sometimes fails when performing TestGocannon
:
=== RUN TestGocannon
assertion_compare.go:342:
Error Trace: integration_test.go:78
Error: "2.1263238042593002" is not less than or equal to "2"
Test: TestGocannon
Messages: [average latencies calculated by reqlog and hist should be within the error margin]
During this test, logic identical to runGocannon()
is performed by load testing a sample HTTP server (target_test.go
). The only difference when compared to an actual run is that performance metrics are collected with both the request log and latency histogram (as well as atomic counters) so as to ensure that all methods obtain equal results (within the specified error margin when calculating the average).
When running the test on GitHub Actions workers, the difference between average request latency obtained via the request log and latency histogram statistic collectors sometimes exceeds the given threshold of 2 nanoseconds. Since the potential discrepancy between the two methods is proportional to the request latencies obtained during the test, the problem is likely caused by GitHub Action workers having less resources than my local setup, on which I initially developed the tests (I'm getting averages of about 0.5ms on local vs ~10ms in CI). In order to fix this issue, the test should be modified so as to calculate the maximum threshold as a percentage of the obtained value (i.e 0.01%), not a fixed 2ns value.