Skip to content

CI action sometimes fails during integration/system test #10

@kffl

Description

@kffl

The CI workflow sometimes fails when performing TestGocannon:

 === RUN   TestGocannon
    assertion_compare.go:342: 
        	Error Trace:	integration_test.go:78
        	Error:      	"2.1263238042593002" is not less than or equal to "2"
        	Test:       	TestGocannon
        	Messages:   	[average latencies calculated by reqlog and hist should be within the error margin]

During this test, logic identical to runGocannon() is performed by load testing a sample HTTP server (target_test.go). The only difference when compared to an actual run is that performance metrics are collected with both the request log and latency histogram (as well as atomic counters) so as to ensure that all methods obtain equal results (within the specified error margin when calculating the average).

When running the test on GitHub Actions workers, the difference between average request latency obtained via the request log and latency histogram statistic collectors sometimes exceeds the given threshold of 2 nanoseconds. Since the potential discrepancy between the two methods is proportional to the request latencies obtained during the test, the problem is likely caused by GitHub Action workers having less resources than my local setup, on which I initially developed the tests (I'm getting averages of about 0.5ms on local vs ~10ms in CI). In order to fix this issue, the test should be modified so as to calculate the maximum threshold as a percentage of the obtained value (i.e 0.01%), not a fixed 2ns value.

Metadata

Metadata

Assignees

No one assigned

    Labels

    CIContinuous Integration

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions