Failsafe TestHTMLReporter parameter.toString #2830
Description
TestNG Version
7.6.2-SNAPSHOT
Expected behavior
Failsafe TestHTMLReporter when parameter.toString() of test execution throws RuntimeException.
Actual behavior
TestHTMLReporter does not fail if test parameter toString method throws Exception. Sequentially it leads to maven build failure with non-clear diagnostics (missing full stacktrace, only message) like
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
...
build 10-Nov-2022 05:44:17 [INFO] ------------------------------------------------------------------------
build 10-Nov-2022 05:44:17 [INFO] BUILD FAILURE
build 10-Nov-2022 05:44:17 [INFO] ------------------------------------------------------------------------
build 10-Nov-2022 05:44:17 [INFO] Total time: 10:19 min (Wall Clock)
build 10-Nov-2022 05:44:17 [INFO] Finished at: 2022-11-10T05:44:17Z
build 10-Nov-2022 05:44:17 [INFO] ------------------------------------------------------------------------
build 10-Nov-2022 05:44:17 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.22.2:verify (default-cli) on project users-integration-test: There are test failures.
build 10-Nov-2022 05:44:17 [ERROR]
build 10-Nov-2022 05:44:17 [ERROR] Please refer to /home/bamboo/bamboo-agent-home/xml-data/build-dir/RTB-NRB22926-JOB10/server/general/users-integration-test/target/failsafe-reports for the individual test results.
build 10-Nov-2022 05:44:17 [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
build 10-Nov-2022 05:44:17 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
build 10-Nov-2022 05:44:17 [ERROR] game_type_id=10000006, model=Model [class=projects.pt.server.users.models.userid.UserIdModel, name=UserId, key=-4268742994815057915:UserId, id=145, serverOnly=true], field=PersistentFieldPersistentModelField[UserIdEntity UserIdModel.userIdEntity]
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:656)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:190)
build 10-Nov-2022 05:44:17 [ERROR] at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:186)
build 10-Nov-2022 05:44:17 [ERROR] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
build 10-Nov-2022 05:44:17 [ERROR] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
build 10-Nov-2022 05:44:17 [ERROR] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
build 10-Nov-2022 05:44:17 [ERROR] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
build 10-Nov-2022 05:44:17 [ERROR] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
build 10-Nov-2022 05:44:17 [ERROR] at java.base/java.lang.Thread.run(Thread.java:833)
build 10-Nov-2022 05:44:17 [ERROR] -> [Help 1]
build 10-Nov-2022 05:44:17 org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.22.2:verify (default-cli) on project users-integration-test: There are test failures.
The good thing is that failure can be found in target/failsafe-reports/*-jvmRun*.dump and there is stacktrace. But if test suite is executed on the CI/CD it can be problem to obtain it.
How it can happen: objects that are parameters of tests may have stateful toString() method (actually antipattern, but failsafe still makes sense to handle such failures).
Is the issue reproducible on runner?
- Shell
- Maven (with HTML reporting)
- Gradle
- Ant
- Eclipse
- IntelliJ
- NetBeans
Test case sample
The PR with the proposed fix #2831
Contribution guidelines
Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.