Junit5 test discovery reports spurious test runs for non-test classes (500USD bounty) #3910
Description
From the maintainer Li Haoyi: I'm putting a 500USD bounty on this issue, payable by bank transfer on a merged PR implementing this.
First noticed when reviewing #3886
lihaoyi mill$ ./mill dist.run example/kotlinlib/basic/1-simple -i foo.test
[2617/2617] dist.run
======================================================== foo.test ====================================================
======================================================================================================================
[99/99] foo.test.test
[99] Warning: Kotest autoscan is enabled. This means Kotest will scan the classpath for extensions that are annotated with @AutoScan. To avoid this startup cost, disable autoscan by setting the system property 'kotest.framework.classpath.scanning.autoscan.disable=true'. In 6.0 this value will default to true. For further details see https://kotest.io/docs/next/framework/project-config.html#runtime-detection
[99] Test run started (Kotest)
[99] Test foo.FooTesttestSimple started
[99] Test foo.FooTesttestSimple finished, took 0.038s
[99] Test foo.FooTesttestEscaping started
[99] Test foo.FooTesttestEscaping finished, took 0.001s
[99] Test foo.FooTest finished, took 0.09s
[99] Test finished, took 0.11s
[99] Test finished, took 0.002s
[99] Test run finished: 0 failed, 0 ignored, 2 total, 0.14s
[99] Warning: Kotest autoscan is enabled. This means Kotest will scan the classpath for extensions that are annotated with @AutoScan. To avoid this startup cost, disable autoscan by setting the system property 'kotest.framework.classpath.scanning.autoscan.disable=true'. In 6.0 this value will default to true. For further details see https://kotest.io/docs/next/framework/project-config.html#runtime-detection
[99] Test finished, took 0.001s
[99] Test finished, took 0.0s
[99] Test run finished: 0 failed, 0 ignored, 0 total, 0.001s
[99] Warning: Kotest autoscan is enabled. This means Kotest will scan the classpath for extensions that are annotated with @AutoScan. To avoid this startup cost, disable autoscan by setting the system property 'kotest.framework.classpath.scanning.autoscan.disable=true'. In 6.0 this value will default to true. For further details see https://kotest.io/docs/next/framework/project-config.html#runtime-detection
[99] Test finished, took 0.0s
[99] Test finished, took 0.0s
[99] Test run finished: 0 failed, 0 ignored, 0 total, 0.001s
[99] Warning: Kotest autoscan is enabled. This means Kotest will scan the classpath for extensions that are annotated with @AutoScan. To avoid this startup cost, disable autoscan by setting the system property 'kotest.framework.classpath.scanning.autoscan.disable=true'. In 6.0 this value will default to true. For further details see https://kotest.io/docs/next/framework/project-config.html#runtime-detection
[99] Test finished, took 0.0s
[99] Test finished, took 0.0s
[99] Test run finished: 0 failed, 0 ignored, 0 total, 0.001s
[99/99] ================================================ foo.test ================================================= 7s
======================================================================================================================
[2617/2617] ======================= dist.run example/kotlinlib/basic/1-simple -i foo.test ============================ 8s
The line Test run finished: 0 failed, 0 ignored, 2 total is correct, but there also are three line Test run finished: 0 failed, 0 ignored, 0 total which are spurious. It seems to be something to do with the test framework incorrectly picking up the anonymous lambda classes as test classes, even though they have no tests
Also happens for non-kotlin tests, e.g. here is a standalone repro using Java/junit5 where the class Dummy causes a spurious Test run finished: 0 failed, 0 ignored, 0 total to be logged
lihaoyi mill$ git diff
diff --git a/example/javalib/testing/3-integration-suite/qux/test/src/qux/QuxTests.java b/example/javalib/testing/3-integration-suite/qux/test/src/qux/QuxTests.java
index 579a2eb55a..fa18297de3 100644
--- a/example/javalib/testing/3-integration-suite/qux/test/src/qux/QuxTests.java
+++ b/example/javalib/testing/3-integration-suite/qux/test/src/qux/QuxTests.java
@@ -16,4 +16,5 @@ public class QuxTests {
String result = Qux.hello();
assertTrue(result.endsWith("World"));
}
-}
\ No newline at end of file
+}
+class Dummy{}
lihaoyi mill$ ./mill dist.run example/javalib/testing/3-integration-suite -i show qux.test
[2617/2617] dist.run
====================================================== show qux.test =================================================
======================================================================================================================
[1/1] show
[1-83/83] qux.test.test
[1-83] Test run started (JUnit Jupiter)
[1-83] Test qux.QuxTests#hello() started
[1-83] Test qux.QuxTests#hello() finished, took 0.012s
[1-83] Test qux.QuxTests#world() started
[1-83] Test qux.QuxTests#world() finished, took 0.001s
[1-83] Test qux.QuxTests finished, took 0.019s
[1-83] Test finished, took 0.027s
[1-83] Test run finished: 0 failed, 0 ignored, 2 total, 0.037s
[1-83] Test finished, took 0.001s
[1-83] Test run finished: 0 failed, 0 ignored, 0 total, 0.001s
[
"",
[
{
"fullyQualifiedName": "qux.QuxTests",
"selector": "hello()",
"duration": 11,
"status": "Success"
},
{
"fullyQualifiedName": "qux.QuxTests",
"selector": "world()",
"duration": 1,
"status": "Success"
}
]
]
[83/83] ============================================== show qux.test ============================================== 1s
[1] show
[1-83] qux.test.test
======================================================================================================================
[83/83] ============================================== show qux.test ============================================== 1s
======================================================================================================================
[2617/2617] =============== dist.run example/javalib/testing/3-integration-suite -i show qux.test ==================== 2s