TestNG Version

7.10.2

Expected behavior

Test method is retried according to IRetryAnalyzer configuration when parameters are modified before rerun

Actual behavior

Test method is retried endlessly when parameters are modified before rerun

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Summary:

In my current project, test data needs to be slightly modified during retries because the data is saved to a database during test execution and on reruns, errors occur due to the data already existing in the database.

To handle this, parameters are modified within the ITestListener.onTestSkipped method during reruns to generate new values for each retry. The object remains the same type, but some fields (e.g., description or UUID-number) are updated.

Issue:

After upgrading TestNG from 7.7.1 to 7.10.2, test methods with updated parameters are invoked endlessly.

While debugging, I traced the issue to the method org.testng.internal.BaseTestMethod.getRetryAnalyzerConsideringMethodParameters(ITestResult tr). In this method, a unique keyAsString is generated for each rerun, which results in the creation of a new IRetryAnalyzer every time.
Corresponding PR: #2935

Additional Info:

I attempted to use DataProvider with cacheDataForTestRetries = false, but it fails to supply new parameters to the test methods.
As a result, I'm currently blocked from upgrading TestNG due to this issue.
Related issue: #3157
Related pr: #3076

Test case sample

Test class

public class EndlessRetryTest {
    Logger logger = Logger.getLogger(getClass().getName());

    @Test(dataProvider = "retryTest")
    public void retryTest(SomeDataEntity someDataEntity) {

        logger.info("Param : " + someDataEntity.getName());
        assert 1 == 0;
    }

    @DataProvider(name = "retryTest")
    public Object[][] getData() {

        return new Object[][]{{new SomeDataEntity()}};
    }

    @Data
    class SomeDataEntity implements Randomizer {
        String name = "Default";

        @Override
        public void randomize() {
            this.name = UUID.randomUUID().toString();
        }
    }
}

RetryListener class

public class RetryListener implements IAnnotationTransformer, ITestListener {
    @Override
    public void transform(ITestAnnotation iTestAnnotation, Class aClass, Constructor constructor, Method method) {

        iTestAnnotation.setRetryAnalyzer(Retrier.class);
    }

    @Override
    public void onTestSkipped(ITestResult result) {

        Object[] parameters = result.getParameters();

        for (Object parameter : parameters) {

            if (parameter instanceof Randomizer) {

                ((Randomizer) parameter).randomize();
            }
        }
    }
}

Retrier class

public class Retrier implements IRetryAnalyzer {
    private final AtomicInteger counter = new AtomicInteger(1);

    @Override
    public boolean retry(ITestResult result) {
        return counter.getAndIncrement() != 3;
    }

}

Randomizer interface

public interface Randomizer {
    void randomize();
}

retry_testng.xml

 <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Retry_Suite" configfailurepolicy="continue" parallel="methods">
    <test verbose="1" name="Retry_Test" enabled="true" thread-count="5">
        <classes>
            <class name="testpackage.EndlessRetryTest">
            </class>
        </classes>
    </test>
    <listeners>
        <listener class-name="testlisteners.RetryListener"/>
    </listeners>
</suite>

The output during the test run is as follows:

Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : Default

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : 4615d8f0-c025-4e06-97d4-07701368d2e4

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : 2636d0e8-6053-47ed-9591-403a234002d7

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : aad2824a-73d5-4622-9c4b-410a72bd892d

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : a2162904-86d1-4fc5-9dba-da9381bcc1bf

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : 37c00a0a-c430-4362-9900-cf24ff0d0b21

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : ad3f9daa-dd6e-4215-9bac-220450d39e65

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : d51f2091-e779-45bf-b27f-a9bb3b3f669e

Test ignored.
Aug 19, 2024 12:34:05 PM testpackage.EndlessRetryTest retryTest
INFO: Param : 4925c9be-aaa1-4249-9c74-722807f485d1

 (and further endless)