How to Execute Failed Test Cases in TestNG

TestNG is a testing framework that simplifies many testing needs. It stands for Test Next Generation, an open-source test automation framework inspired by JUnit and NUnit. Think of TestNG as an updated version of the other two concepts. It provides additional features not previously available, such as test annotations in code, grouping, prioritization, benchmarking, and process analysis. The TestNG framework not only manages cases but also provides detailed information about these tests. Provides detailed information about the number of failed tests. The report also allows people to test for diseases and treat them at an early stage.

Method to Execute Failed Test Cases in TestNG

1. Use the TestNG XML Configuration File

Create a new TestNG XML file or modify an existing file to include the “FailedReporter” listener and set the “rerun failed” property.

2. Execute the Test Suite

Execute the test suite using TestNG and this modified XML file.

testng -testname testng.xml

TestNG will run the test and create a file called testng-failed.xml containing the list of failed tests.

3. Rerun Failed Tests

TestNG will create a file named `testng-failed.xml` containing the failed tests. You can now retry only failed attempts using the same XML file.

testng -testname testng-failed.xml

TestNG only executes failed tests specified in the `testng-failed.xml` file.

This method will help in repeating only the failed tests instead of redoing the entire package.

How to Execute Failed Test Cases in TestNG?

There are two ways to perform a failed test in TestNG:

1. Using the testng-failed.xml file.
2. Using the IRetryAnalyzer interface.

1. Failed Test Re-run (by using testng-failed.xml)

1. Automatic test run after first start-up. Right click on the object – click Refresh.
2. A list named “test-output” will be created. You can find the “testng-failed.xml” file in the “test-output” folder.
3. To rerun the failed test, run “testng-failed.xml”.

Below is the sample program:

Java

//Java Program for Failed test re-run by using testng-failed.xml

import org.testng.Assert;
import org.testng.annotations.Test;
public class OnlyFailedTestExecution {
 @Test
    public void testOne() {
        Assert.assertTrue(true);
        System.out.println("Pass Test Case");
    }
@Test
    public void testTwo() {
        Assert.assertTrue(false);
        System.out.println("Faile Test Case");
    }
}

Execute the above program using below testng.xml file:

XML

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
  <test name="Test">
    <classes>
      <class name="OnlyFailedTestExecution"/>
    </classes>
  </test>
</suite>

After processing the above xml file we can see the following output:

We can find the output

Now we just want the test to fail. To do this, we will go to the test-failed folder and create the testng-failed.xml file. We can then see the following output.

The test-output folder

In the above output we can see that only failed test got executed. And it ignores the execution of passed tests.

2. Failed Test Re-run (by implementing IRetryAnalyzer)

1. Create a class to implement TestNG IRetryAnalyzer interface:

Java

//Java program for Failed test re-run by implementing IRetryAnalyzer
public interface IRetryAnalyzer {
 /**
   * Returns true if the test method has to be retried, false otherwise.
   * @param result The result of the test method that just ran.
   * @return true if the test method has to be retried, false otherwise.
   */
  public boolean retry(ITestResult result);
}

This interface has only one method: public boolean Repeat(ITestResult result);

• This method will be called when the test method fails. You can get test results from the estResult input parameter of this method as shown in the method definition above.
• This method should return true if you want to rerun the failed test, or false if you do not want to rerun the test. The most common use of this dialog is to determine how many times to retry a failed test based on a fixed counter or complex logic based on your needs.

An easy-to-use interface is shown below.

Java

package Tests;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class RetryAnalyzer implements IRetryAnalyzer {
    int counter = 0;
    int retryLimit = 4;
    /*
     * (non-Javadoc)
     * @see org.testng.IRetryAnalyzer#retry(org.testng.ITestResult)
     * This method decides how many times a test needs to be rerun.
     * TestNg will call this method every time a test fails. So we
     * can put some code in here to decide when to rerun the test.
     * Note: This method will return true if a tests needs to be retried
     * and false it not.
     */
@Override
    public boolean retry(ITestResult result) {
        if(counter < retryLimit)
        {
            counter++;
            return true;
        }
        return false;
    }
}

We now have a simple implementation of IRetryAnalyzer. Note the implementation of the retry method, which allows the failed test to be repeated 4 times. This is because we set retryLimit = 4;

Let’s see how to use it, there are two ways to include retry parameters in the test

• By specifying retryAnalyzer value in the @Test annotation
• By adding Retry analyser during run time by implementing on the of the Listener interfaces

2. In Test Method specify Retry Class:

Specifying the value of retryAnalyzer in the @Test annotation. We can do this using the following @Test(retryAnalyzer = “IRetryAnalyzer Implementation class”) expression.

Java

import org.testng.Assert;
import org.testng.annotations.Test;
public class Test001 {
    @Test(retryAnalyzer = Tests.RetryAnalyzer.class)
    public void Test1()
    {
        Assert.assertEquals(false, true);
    }
    @Test
    public void Test2()
    {
        Assert.assertEquals(false, true);
    }
}

We have two tests here; one tells which category to use as the repeater. If we do this test we get the following results.

Specifying which class to use as retry analyser.

Test1 was run 4 times and was marked as failed only on the last run. The counter was run 4 times due to repetition. However, when we look at Test2, we can see that it only runs once.

3. By adding Retry analyser during runtime by implementation of the listener interface.

• The ITestAnnotationTransformer interface belongs to a large class of interfaces called TestNG Listeners.
• This interface is used to programmatically annotate test methods at runtime. It calls each test variable method during the test run.
• When we use IAnnotationTransformer, we add it as a listener to the TestNG runtime xml.

Java

public class AnnotationTransformer implements IAnnotationTransformer {
    @Override
    public void transform(ITestAnnotation annotation,
                                                Class testClass,
                                                Constructor testConstructor,
                                                Method testMethod) {
        // TODO Auto-generated method stub
    }
}

This interface is used to programmatically annotate test methods at runtime. It calls any attempt to change the path during the test run. An easy-to-use interface helps us create back testing tools.

Java

package Tests;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
public class AnnotationTransformer implements IAnnotationTransformer {
    @Override
    public void transform(ITestAnnotation annotation,
                                                Class testClass,
                                                Constructor testConstructor,
                                                Method testMethod) {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

After implementing IAnotationTransformer, we just need to add it as a listener in the xml test run. like this

XML

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="RetryFailedTests" verbose="1" >
 <listeners>
        <listener class-name="Tests.AnnotationTransformer"/>
  </listeners>
  <test name="RetryMulitple">
    <classes>
      <class name="Tests.Test001"/>
    </classes>
  </test>
</suite>

We no longer need to specify the retryAnalyzer attribute in the @Test annotation. The new test will look like this

Java

package Tests;
import org.testng.Assert;
import org.testng.annotations.Test;
public class Test001 {
    @Test
    public void Test1()
    {
        Assert.assertEquals(false, true);
    }
    @Test
    public void Test2()
    {
        Assert.assertEquals(false, true);
    }
}

Now you can complete the test and see how TestNG re-executed each of Test1 and Test2 4 times.

Conclusion

Resolving failing test cases is an important part of the testing process. TestNG makes it easy to recover from failed tests by providing easy-to-use methods via the testng-failed.xml file and built-in listeners. Using these resources, testers and developers can effectively identify and resolve issues, thus ensuring the stability and reliability of their testing units. By following the steps outlined in this guide, testers can manage and execute failed tests with TestNG, helping improve test performance and develop better software.