How to run parallel test cases in TestNG

Selenium is a widely used tool for the automated testing of web applications. It is open-source, supports many programming languages, such as Java, Python, Ruby, C#, and Javascript, and works across multiple operating systems and browsers, such as Chrome, Firefox, IE, and Edge. However, one of the missing features of Selenium is that it does not have any reporting mechanism

What is reporting in Selenium, and Why is it required?

Reporting is necessary for manual as well as automated testing. Without creating good summary report one cannot showcase the health of the application under test. Reports give a statistic of passed, failed, and skipped test cases. In Selenium automation reports, along with pass/fail/skipped statistics, QAs would require also some more details of test automation metrics like total execution time, start and end time of execution, screenshots, text, video, and console logs of failed test cases, environment details of the test execution, etc.

To produce such reports with this extensive data, it is required to integrate Selenium tests with a reporting library. There are many reporting libraries available, out of which TestNG is one of the most widely used.

What is TestNG?

TestNG is a testing framework inspired by JUnit and NUnit but introduces some new functionalities that make it more powerful and easier to use. TestNG helps organize the tests and produce the test reports. You need to integrate this by adding the TestNG library to the automation framework. After adding the TestNG library, you can easily add the necessary TestNG annotations and execute the test automation script.

TestNG helps to efficiently organise the tests and maintain readability of the test cases. TestNG makes it easy to perform parallel test execution by defining parallel attributes in testing.xml file.

What is Parallel Execution in TestNG?

Parallel testing is a process where multiple tests are executed simultaneously/in parallel in different thread processes. With respect to Selenium and TestNG, it allows you to execute multiple tests on different browsers, devices, environments in parallel and at the same time, instead of running it sequentially.

The main purpose of running tests in parallel mode is to reduce execution time and do maximum environment coverage (browsers/devices/environment) in less time.

Suppose, for an application you need to execute a sanity automation suite of 50 test cases in Chrome and Firefox browser. If you go with the traditional sequential flow, you need to execute the suite for Chrome browser first which would take 1 hr and then you need to execute for Firefox browser which takes another 1 hr. So, in total you would need 2 hrs to test in both the browsers. By using a parallel mechanism, you can run simultaneously for both the browsers in just 1 hr thereby reducing the execution time by 50%.

TestNG helps to run test methods/classes/tests in parallel. Using the testng.xml file, one can specify parallel attributes to classes, tests, and methods. Java’s multi-thread feature can also be applied by defining the number of threads for parallel testing in the thread attribute.

<suite name="Parallel Test Suite" thread-count="2" parallel="methods" >

Download required Maven dependencies

1. Install Java 8 or higher and set JAVA_HOME in system environment variables.
2. For a Maven project, add Selenium Java, TestNG and WebDriverManager dependencies. Save the pom.xml file to download all the dependencies.
3. Add the TestNG library to classpath.

<dependencies>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.5.0</version>
</dependency>

<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.6.1</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.github.bonigarcia</groupId>
<artifactId>webdrivermanager</artifactId>
<version>5.2.1</version>
</dependency>
</dependencies>

Executing Test Methods Sequentially in TestNG

Below program runs the test methods sequentially:

Step 1 Under src/test/java create a package and under that create a class as ParallelTest

public class ParallelTest {

WebDriver driver;

@Test(priority = 1)
public void testChrome() throws InterruptedException {
System.out.println("The thread ID for Chrome is "+ Thread.currentThread().getId());
WebDriverManager.chromedriver().setup();
driver = new ChromeDriver();
driver.get("https://www.bstackdemo.com/");
driver.manage().window().maximize();
Assert.assertEquals(driver.getTitle(), "StackDemo");
}

@Test(priority = 2)
public void testFirefox() throws InterruptedException {
System.out.println("The thread ID for Firefox is "+ Thread.currentThread().getId());
WebDriverManager.firefoxdriver().setup();
driver = new FirefoxDriver();
driver.get("https://www.bstackdemo.com/");
driver.manage().window().maximize();
Assert.assertEquals(driver.getTitle(), "StackDemo"); 
}

@AfterClass
public void close() {
driver.quit();
} 
}

Step 2 Right click on the class and select Run As >> TestNG Test. Observe the time taken to execute both the methods in a sequential manner. It takes 18486 ms as seen below

Now let us run both the methods in parallel.

To do so you need to first create a testing.xml file and add a parallel attribute for the test suite with value as methods.

Parallel Execution in TestNG

TestNG allows you Parallel Execution at different levels such as:

• Parallel Execution of Test Methods
• Parallel Execution of Test Classes and
• Parallel Execution of Test Suites

You can understand these different levels of Parallel Execution in TestNG using examples as demonstrated in the sections below.

Executing Parallel Test Methods in TestNG

Step 1 To create a testing.xml file, right click on the ParallelTest class and select TestNG >> Convert To TestNG.

Step 2 You may select Parallel mode and ThreadCount value of your choice while creating the testing.xml file or you may update it later as per the requirement change. I have selected Parallel mode as methods and ThreadCount as 2.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Parallel Test Suite" parallel="methods" thread-count="2">
<test name="Parallel Test" >
<classes>
<class name="com.qa.testcases.ParallelTest"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->

Step 3 Right click on the testing.xml file and select “Run As” -> “TestNG Suite”. Observe the time taken to execute both the methods in parallel mode(10204 ms, decreased the execution timeby 8282 ms).

Executing Test Classes in Parallel using TestNG

Step 1 To run classes in parallel mode, create two class files as ChromeTest and FirefoxTest with three test methods.

public class ChromeTest {
WebDriver driver;

@BeforeTest
public void setUp() {
WebDriverManager.chromedriver().setup();
driver = new ChromeDriver();
driver.get("https://www.bstackdemo.com/");
driver.manage().window().maximize();
}

@Test(priority = 1)
public void testTitle() {
System.out.println("The thread ID for testTitle Chrome is "+ Thread.currentThread().getId());
Assert.assertEquals(driver.getTitle(), "StackDemo");
}

@Test(priority = 2)
public void clickOffers() throws InterruptedException {
System.out.println("The thread ID for clickOffers Chrome is "+ Thread.currentThread().getId());
WebElement offers=driver.findElement(By.cssSelector("a#offers"));
offers.click();
Thread.sleep(2000);
WebElement loginBtn=driver.findElement(By.cssSelector("button#login-btn"));
Assert.assertTrue(loginBtn.isDisplayed()); 
}

@Test(priority = 3)
public void clickOrders() throws InterruptedException {
driver.navigate().to("https://www.bstackdemo.com/");
System.out.println("The thread ID for clickOrders Chrome is "+ Thread.currentThread().getId());
WebElement orders=driver.findElement(By.cssSelector("a#orders"));
orders.click();
Thread.sleep(2000);
WebElement loginBtn=driver.findElement(By.cssSelector("button#login-btn"));
Assert.assertTrue(loginBtn.isDisplayed());
}

@AfterTest
public void tearDown() {
driver.close();
}
}


public class FirefoxTest {
WebDriver driver;

@BeforeTest
public void setUp() {
WebDriverManager.firefoxdriver().setup();
driver = new FirefoxDriver(); 
driver.get("https://www.bstackdemo.com/");
driver.manage().window().maximize();
}

@Test(priority = 1)
public void testTitle() { 
System.out.println("The thread ID for testTitle Firefox is " + Thread.currentThread().getId());
Assert.assertEquals(driver.getTitle(), "StackDemo");
}

@Test(priority = 2)
public void clickOffers() throws InterruptedException {
System.out.println("The thread ID for clickOffers Firefox is " + Thread.currentThread().getId());
WebElement offers = driver.findElement(By.cssSelector("a#offers"));
offers.click();
Thread.sleep(3000);
WebElement loginBtn = driver.findElement(By.cssSelector("button#login-btn")); 
Assert.assertTrue(loginBtn.isDisplayed());
}

@Test(priority = 3)
public void clickOrders() throws InterruptedException {
driver.navigate().to("https://www.bstackdemo.com/");
System.out.println("The thread ID for clickOrders Firefox is "+ Thread.currentThread().getId());
WebElement orders=driver.findElement(By.cssSelector("a#orders"));
orders.click();
Thread.sleep(2000);
WebElement loginBtn=driver.findElement(By.cssSelector("button#login-btn"));
Assert.assertTrue(loginBtn.isDisplayed());
}
@AfterTest
public void tearDown() {
driver.quit();
}

}

Step 2 In the testing.xml file, add class names and update parallel value as classes and run it.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Parallel Test Suite" parallel="classes" thread-count="2">
<test name="Parallel Test" >
<classes>
<class name="com.qa.testcases.ChromeTest"/>
<class name="com.qa.testcases.FirefoxTest"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->

To demonstrate that two classes are running on different threads I have added a code to print the current Thread Id. Note that ChromeTest is running on Thread ID 14 and FirefoxTest is on Thread ID 15.

Obviously, as demonstrated in parallel methods, execution time while running classes in parallel is lesser when classes are run sequentially.

Executing Test Suites in Parallel using TestNG

To run all the tests available inside the suite tag in parallel mode, you need to update the “parallel” value as “tests” in testing.xml file.

Here thread-count value plays an important role because if thread count is less than tests, tests need to wait for other tests to execute.

Let us understand this by following example:

Step 1 Create one more class as EdgeTest. ChromeTest, FirefoxTest, and EdgeTest classes have 3 test methods.

public class EdgeTest {
WebDriver driver;

@BeforeTest
public void setUp() {
WebDriverManager.edgedriver().setup();
driver = new EdgeDriver();
driver.get("https://www.bstackdemo.com/");
driver.manage().window().maximize();
}

@Test(priority = 1)
public void testTitle() {
System.out.println("The thread ID for testTitle Edge is "+ Thread.currentThread().getId());
Assert.assertEquals(driver.getTitle(), "StackDemo");
}

@Test(priority = 2)
public void clickOffers() throws InterruptedException {
System.out.println("The thread ID for clickOffers Edge is "+ Thread.currentThread().getId());
WebElement offers=driver.findElement(By.cssSelector("a#offers"));
offers.click();
Thread.sleep(2000);
WebElement loginBtn=driver.findElement(By.cssSelector("button#login-btn"));
Assert.assertTrue(loginBtn.isDisplayed()); 
}

@Test(priority = 3)
public void clickOrders() throws InterruptedException {
driver.navigate().to("https://www.bstackdemo.com/");
System.out.println("The thread ID for clickOrders Edge is "+ Thread.currentThread().getId());
WebElement orders=driver.findElement(By.cssSelector("a#orders"));
orders.click();
Thread.sleep(2000);
WebElement loginBtn=driver.findElement(By.cssSelector("button#login-btn"));
Assert.assertTrue(loginBtn.isDisplayed());
}

@AfterTest
public void tearDown() {
driver.close();
}
}

Step 2 Update parallel value as tests and add all the 3 tests in testing.xml file. Keep thread count as 2 and run the testing.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Parallel Test Suite" thread-count="2" parallel="tests" >
<test name="Parallel Test Chrome" >
<classes>
<class name="com.qa.testcases.ChromeTest"/>
</classes>
</test> <!-- Test -->

<test name="Parallel Test Firefox" >
<classes>
<class name="com.qa.testcases.FirefoxTest"/>
</classes>
</test> <!-- Test -->

<test name="Parallel Test Edge" >
<classes>
<class name="com.qa.testcases.EdgeTest"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->

When thread count is set to 2, ChromeTest and FireTest will execute in parallel on 2 Threads. EdgeTest will wait for either one of them to execute so that it can execute on the freed thread.

You can see from the above console logs, that “FirefoxTest” and “ChromeTest” ran in parallel on Thread 14 and 15 respectively. Later “EdgeTest” ran on Thread 15 as it was freed earlier than 14.

If thread count would have been “3”, all three tests would have run on Thread 14, 15 and 16 respectively in parallel thereby decreasing execution time.

Talk to an Expert

Cons of parallel testing using TestNG:

1. Parallel testing is considerate while testing independent modules/classes. It fails for modules which are dependent on another module thereby giving flaky results.
2. Tester should have a detailed understanding of the product under test and the workflow of the testcases to apply parallelism. If any dependent module is run in parallel mode, complete test execution may go for a toss.

Conclusion

Sequential testing is time consuming and therefore, parallel testing is required to decrease the execution time and cover more devices, browsers, platforms under test. With enhancement and innovations in the technology field, companies are frequently launching new devices, browser/ updated browsers and platforms to give a better user experience. It becomes challenging for a tester to test against all such combinations and on the other hand it is obligatory to test that too.

BrowserStack offers 3000+ real devices and browsers to help achieve cross browser testing in parallel using Selenium through its Cloud Selenium Grid. Sign up today to have a seamless experience while running parallel tests!

Run Parallel Tests on BrowserStack