panic: Log in goroutine after TestScenarioPathFilterDeny has completed #1227
Description
This panic happens indefinitely when running the test TestScenarioPathFilterDeny.
It should be caused by zap logger error logging after test completed inside cosmos.(*CosmosChainProcessor).Run, starting from a goroutine which is out of control.
In cosmos_chain_processor.go, queryCycle is called within an infinite loop when context was not cancelled. It can cause the panic issue if context cancel is not detected before a queryCycle call, while the test is going to complete.
2023-07-03T01:32:42.3131993Z panic: Log in goroutine after TestScenarioPathFilterAllow has completed: 2023-07-03T01:32:41.435Z DEBUG No cached IBC header for client trusted height {"chain_id": "osmosis-12", "client_id": "07-tendermint-0", "height": 34}
2023-07-03T01:32:42.3132065Z
2023-07-03T01:32:42.3132231Z goroutine 12832 [running]:
2023-07-03T01:32:42.3132441Z testing.(*common).logDepth(0xc001402ea0, {0xc00065cbe0, 0x96}, 0x3)
2023-07-03T01:32:42.3132863Z /opt/hostedtoolcache/go/1.21.0-rc2/x64/src/testing/testing.go:1022 +0x674
2023-07-03T01:32:42.3132995Z testing.(*common).log(...)
2023-07-03T01:32:42.3133302Z /opt/hostedtoolcache/go/1.21.0-rc2/x64/src/testing/testing.go:1004
2023-07-03T01:32:42.3133508Z testing.(*common).Logf(0xc001402ea0, {0x5b87048, 0x2}, {0xc000b12b60, 0x1, 0x1})
2023-07-03T01:32:42.3133848Z /opt/hostedtoolcache/go/1.21.0-rc2/x64/src/testing/testing.go:1055 +0x9a
2023-07-03T01:32:42.3134119Z go.uber.org/zap/zaptest.testingWriter.Write({{0x5bc5270?, 0xc001402ea0?}, 0x80?}, {0xc0014ae400, 0x97, 0x400})
2023-07-03T01:32:42.3134363Z /home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/zaptest/logger.go:130 +0x11e
2023-07-03T01:32:42.3134673Z go.uber.org/zap/zapcore.(*ioCore).Write(0xc000fba960, {0xff, {0xc120a72e59fae504, 0xbe9efc76b0, 0x745e480}, {0x0, 0x0}, {0x4131bfb, 0x2e}, {0x0, ...}, ...}, ...)
2023-07-03T01:32:42.3134903Z /home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/core.go:99 +0x193
2023-07-03T01:32:42.3135117Z go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00102d930, {0xc001c503c0, 0x3, 0x3})
2023-07-03T01:32:42.3135342Z /home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:255 +0x2b0
2023-07-03T01:32:42.3135560Z go.uber.org/zap.(*Logger).Debug(0xc000d116c0?, {0x4131bfb, 0x2e}, {0xc001c503c0, 0x3, 0x3})
2023-07-03T01:32:42.3135774Z /home/runner/go/pkg/mod/go.uber.org/zap@v1.24.0/logger.go:212 +0x68
2023-07-03T01:32:42.3136115Z github.com/cosmos/relayer/v2/relayer/processor.(*PathProcessor).updateClientTrustedState(0xc000e6f580, 0xc001cf5000, 0xc001cf4e00)
2023-07-03T01:32:42.3136383Z /home/runner/work/relayer/relayer/relayer/processor/path_processor_internal.go:647 +0x765
2023-07-03T01:32:42.3136716Z github.com/cosmos/relayer/v2/relayer/processor.(*PathProcessor).processLatestMessages(0xc000e6f580, {0x5bbf318?, 0xc001631540}, 0xc0022ba0f0?)
2023-07-03T01:32:42.3136977Z /home/runner/work/relayer/relayer/relayer/processor/path_processor_internal.go:855 +0x105
2023-07-03T01:32:42.3137260Z github.com/cosmos/relayer/v2/relayer/processor.(*PathProcessor).Run(0xc000e6f580, {0x5bbf318, 0xc001631540}, 0xc0022ba0f0)
2023-07-03T01:32:42.3137492Z /home/runner/work/relayer/relayer/relayer/processor/path_processor.go:369 +0x239
2023-07-03T01:32:42.3137750Z github.com/cosmos/relayer/v2/relayer/processor.EventProcessor.Run.func1()
2023-07-03T01:32:42.3137987Z /home/runner/work/relayer/relayer/relayer/processor/event_processor.go:91 +0x55
2023-07-03T01:32:42.3138475Z golang.org/x/sync/errgroup.(*Group).Go.func1()
2023-07-03T01:32:42.3138857Z /home/runner/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x77
2023-07-03T01:32:42.3139075Z created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 12831
2023-07-03T01:32:42.3139307Z /home/runner/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0x125
2023-07-03T01:32:42.3139500Z FAIL github.com/cosmos/relayer/v2/interchaintest 818.751s