How to use TestNG Reporter Log in Selenium: Tutorial

Test reports are critical to a testing process, This article provides a step-by-step guide on using TestNG Reporter and automating it using Selenium.

What is TestNG?

TestNG is an open-source test automation framework created by Cedric Beust, where ‘NG’ stands for Next Generation. TestNG is similar to JUnit, however, it is preferred over JUnit, particularly while testing coordinated classes. It is because TestNG offers ease of using multiple annotations, grouping, dependence, prioritization, and parametrization. 

TestNG allows QAs to categorize or prioritize the test cases, induce HTML reports, log dispatches, run tests in parallel, and perform many more actions. These features help QA leverage TestNG with Selenium in automation testing.

In order to use TestNG Reporter Log, it is necessary to understand the significance of reporting in a test.

Why is Reporting important?

Reports can be insightful and help in deciding the future course of action as it a one-stop source of all the accessible information. With automated report generation frameworks, it enhances the visualization of test results for the betterment of overall analysis.

However, despite sharing all the major information regarding the tests, TestNG reports might at times fail to convey the exact root of failure due to inadequate logging. As a result, you might need to run the whole class to find the problem.

To overcome this problem TestNG provides an inbuilt class called the Reporter Class that is used for logging. This tutorial explores how to use the TestNG Reporter Log in Selenium to identify the cause of a failed test which would improve debugging. 

What is Reporter Class in TestNG

Reporter Class is an inbuilt class in TestNG, which is available under the org.testng package. This class provides test styles to log dispatches that will be included in the HTML reports generated by TestNG. Reporter Class is one of the simplest ways of generating log information, where the logs in the reports can be either stoner-generated or system-generated reports.

TestNG Reporting Class is quite useful as it helps analyze failed tests based on the detailed information from the logs. This avoids the need to rerun the entire test case. By specifying different logs at each step, QAs can use this classification when debugging the failures, making debugging easier.

How to use TestNG Reporter Log in Selenium

Selenium WebDriver is widely used for test automation of websites. However, it lacks reporting of the tests logs, hence TestNG is used to create report logs that would help identify and debug failed tests. TestNG induces dereliction of HTML reports once the test cases are executed. 

Following TestNG Reports can be created:

• emailable-report.html
• index.html
• Reporter Class

To understand how TestNG report works using an example, here is a sample Selenium test with TestNG Report. 

Step 1: Create a Selenium Test Class named as EmailReport and TestNG.xml file using the following code

package testngpackage;

import org.testng.Assert;
import org.testng.SkipException;
import org.testng.annotations.Test;

public class EmailReport {

//To make it pass
@Test
public void passTest(){
Assert.assertTrue(true);
}

//To make it fail
@Test
public void failTest(){
Assert.assertTrue(false);
}

//To make it skip
@Test
public void skipTest(){
throw new SkipException("Skipping -This method is skipped testing ");
}
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<suite name="testngpackage" parallel="methods">
<test name="testngTest">
<classes>
<class name="testngpackage.EmailReport" />
</classes>
</test>
</suite>

Step 2: Execute the testng.xml file and refresh the project. You can see the testng.xml file within the project folder TestNGProject as seen below.

Step 3: Expand test-output folder and you should find a default generated TestNG report emailable-report.html and index.html. Open the report in the browser. Reports will be seen as below:

Emailable Report in TestNG

HTML Index Report in TestNG

Reporter Class in TestNG

In addition to the above two approaches to induce reports, you can also employ the object.properties queue to store the system-generated logs as well as user-generated logs.

Exploiting Reporter Class is one of the easiest ways to reposit log data in testing. Reporter class is an inbuilt class in TestNG. It helps in depositing the logs inside the reports, which are user-generated or system-generated so that in the future when you access the report, you can directly view the logs from there rather than rerunning the test cases.

Reporter is a class present in TestNG. It has four different approaches to storing the log information:

• Reporter.log( String s);
• Reporter.log ( String s, Boolean logToStandardOut);
• Reporter.log ( String s, int level);
• Reporter.log ( String s, int level, Boolean logToStandardOut);

How to log messages in reports generated by TestNG Reporter Class

Below is the sample test to generate logs using TestNG Reporter Class in Selenium Test.

Upon executing the below test, the report will have the following messages logged:

• Browser Opened
• Browser Maximized
• Application Started
• Application Closed

package testngDemo;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.Reporter;
import org.testng.annotations.Test;

public class ReporterDemo {

@Test
public void testReport(){

WebDriver driver=new FirefoxDriver();

Reporter.log("Browser Opened");

driver.manage().window().maximize();

Reporter.log("Browser Maximized");

driver.get("http://www.google.com");

Reporter.log("Application started");

driver.quit();

Reporter.log("Application closed"); 
}
}

Let’s explore more realistic use by creating a class GoogleTest and write the following code inside the class.

package com.sampletestpackage;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.Reporter;
//import org.openqa.selenium.firefox.FirefoxDriver;
//import org.openqa.selenium.ie.InternetExplorerDriver;
import org.testng.annotations.Test;

public class GoogleTest {

WebDriver driver;
@Test(priority = 1) 
public void driverSetup()
{ 
// System.setProperty("webdriver.gecko.driver", "src\\main\\java\\com\\browserdrivers\\geckodriver.exe");
System.setProperty("webdriver.chrome.driver", "src\\main\\java\\com\\browserdrivers\\chromedriver.exe");
//System.setProperty("webdriver.ie.driver", "src\\main\\java\\com\\browserdrivers\\IEDriverServer.exe");

driver=new ChromeDriver(); 
Reporter.log("The browser is opened");

} 
@Test(priority = 2) 
public void getURL()
{ 
driver.get("https://www.google.com");
// System.out.println("Launching Google site"); 
Reporter.log("The Google Site is Launched");

} 
@Test(priority = 3) 
public void getTitle()
{ 
String title = driver.getTitle(); 
System.out.println(title); Reporter.log("Prints the web page title");


} 
@Test(priority = 4) 
public void closeBrowser()
{ 
driver.close(); 
//System.out.println("Test successfully passed"); 
Reporter.log("Close the driver");

} 

}

Create TestNG.xml by selecting the above Test Class and Run the test using TestNG.xml. Open the Reports generated.

The Reports are generated as seen below:

Emailable Report with TestNG Reporter Log

HTML Index Report with TestNG Reporter Log

To help further, check out the below list of most commonly used Reporter Logs snippets

/**
* Log the passed string to the HTML reports
* @param s The message to log
*/
public static void log(String s) {
log(s, getCurrentTestResult());
}


/**
* Log the passed string to the HTML reports. If logToStandardOut
* is true, the string will also be printed on standard out.
*
* @param s The message to log
* @param logToStandardOut Whether to print this string on standard
* out too
*/
public static void log(String s, boolean logToStandardOut) {
log(s, getCurrentTestResult());
if (logToStandardOut) {
System.out.println(s);
}
}

/**
* Log the passed string to the HTML reports if the current verbosity
* is equal or greater than the one passed in parameter
*
* @param s The message to log
* @param level The verbosity of this message
*/
public static void log(String s, int level) {
if (TestRunner.getVerbose() >= level) {
log(s, getCurrentTestResult());
}
}

/**
* Log the passed string to the HTML reports if the current verbosity
* is equal or greater than the one passed in parameter. If logToStandardOut
* is true, the string will also be printed on standard out.
*
* @param s The message to log
* @param level The verbosity of this message
* @param logToStandardOut Whether to print this string on standard
* out too
*/
public static void log(String s, int level, boolean logToStandardOut) {
if (TestRunner.getVerbose() >= level) {
log(s, getCurrentTestResult());
if (logToStandardOut) {
System.out.println(s);
}
}
}


// ReporterSample.report()
@Test(dataProvider = "dp", timeOut = 10000)
public void report(String p) {
Reporter.log("IN THE REPORTER: " + p);
}
}

// ReporterLogListener.onTestFailure()
@Override
public void onTestFailure(ITestResult result) {
Reporter.log("Listener: onTestFailure");
super.onTestFailure(result);
}

// ReporterLogListener.onTestSuccess()
@Override
public void onTestSuccess(ITestResult result) {
Reporter.log("Listener: onTestSuccess");
super.onTestSuccess(result);
}

// ReporterLogListener.onTestSkipped()
@Override
public void onTestSkipped(ITestResult result) {
Reporter.log("Listener: onTestSkipped");
super.onTestSkipped(result);
}
}

How to use TestNG Reporter Log for Parallel Tests using BrowserStack

Testing on a real device cloud allows the QAs to test under real user conditions as opposed to testing on emulators and simulators. This helps in getting more accurate results for the test and identifying bottlenecks in the actual user experience. BrowserStack gives us access to a cloud Selenium Grid of 3000+ real devices and browsers. Running Selenium tests with TestNG on BrowserStack is quite easy as seen in the steps below:

Step 1: Defining a class that contains the methods to configure and establish the connection with BrowserStack.

package testngDemo;

import com.browserstack.local.Local;
import org.json.simple.parser.JSONParser;

public class BrowserStackTestNGTest {
public WebDriver driver;
private Local l;

@BeforeMethod(alwaysRun=true)
@org.testng.annotations.Parameters(value={"config", "environment"})
public void setUp(String config_file, String environment) throws Exception {
JSONParser parser = new JSONParser();
JSONObject config = (JSONObject) parser.parse(new FileReader("src/test/resources/conf/" + config_file));
JSONObject envs = (JSONObject) config.get("environments");

DesiredCapabilities capabilities = new DesiredCapabilities();

Map<String, String> envCapabilities = (Map<String, String>) envs.get(environment);
Iterator it = envCapabilities.entrySet().iterator();
while (it.hasNext()) {
Map.Entry pair = (Map.Entry)it.next();
capabilities.setCapability(pair.getKey().toString(), pair.getValue().toString());
}

Map<String, String> commonCapabilities = (Map<String, String>) config.get("capabilities");
it = commonCapabilities.entrySet().iterator();
while (it.hasNext()) {
Map.Entry pair = (Map.Entry)it.next();
if(capabilities.getCapability(pair.getKey().toString()) == null){
capabilities.setCapability(pair.getKey().toString(), pair.getValue().toString());
}
}

String username = System.getenv("BROWSERSTACK_USERNAME");
if(username == null) {
username = (String) config.get("user");
}

String accessKey = System.getenv("BROWSERSTACK_ACCESS_KEY");
if(accessKey == null) {
accessKey = (String) config.get("key");
}

String app = System.getenv("BROWSERSTACK_APP_ID");
if(app != null && !app.isEmpty()) {
capabilities.setCapability("app", app);
}

if(capabilities.getCapability("browserstack.local") != null && capabilities.getCapability("browserstack.local") == "true"){
l = new Local();
Map<String, String> options = new HashMap<String, String>();
options.put("key", accessKey);
l.start(options);
}

driver = new RemoteWebDriver(new URL("https://"+username+":"+accessKey+"@"+config.get("server")+"/wd/hub"), capabilities);
}

@AfterMethod(alwaysRun=true)
public void tearDown() throws Exception {
driver.quit();
if(l != null) l.stop();
}
}

Step 2: Creating single.conf.json for configuring your BrowserStack user credentials using the code snippet below

{
"server": "hub-cloud.browserstack.com",
"user": "YOUR_USERNAME",
"key": "YOUR_ACCESS_KEY",

"capabilities": {
"build": "testng-browserstack",
"name": "single_test",
"browserstack.debug": true
},

"environments": {
"chrome": {
"browser": "chrome"
}
}
}

Step 3: Creating single.conf.xml file for configuring TestNG using the code snippet below

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Single">
<test name="SingleTest">
<parameter name="config" value="single.conf.json"/>
<parameter name="environment" value="chrome"/>
<classes>
<class name="com.browserstack.SingleTest"/>
</classes>
</test>
</suite>

BrowserStack provides an integral REST API to pierce and modernize facts about our tests, which can be used to mark the tests as pass or fail.

Tag the tests as pass or fail grounded on the assertions in the TestNG test cases using the code below.

// Method to mark test as pass / fail on BrowserStack

public static void mark() throws URISyntaxException, UnsupportedEncodingException, IOException {
URI uri = new URI("https://YOUR_USERNAME:YOUR_ACCESS_KEY@api.browserstack.com/automate/sessions/<session-id>.json");
HttpPut putRequest = new HttpPut(uri);

ArrayList<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
nameValuePairs.add((new BasicNameValuePair("status", "completed")));
nameValuePairs.add((new BasicNameValuePair("reason", "")));
putRequest.setEntity(new UrlEncodedFormEntity(nameValuePairs));

HttpClientBuilder.create().build().execute(putRequest);
}

No matter the software, Selenium WebDriver tests must be executed on real devices and browsers. Remember that device fragmentation is a major concern for every developer and tester. Every website has to work seamlessly on multiple device-browser-OS combinations. With 9000+ distinct devices being used to access the internet globally, all software has to be optimized for different configurations, viewports, and screen resolutions.

In this state, no emulator or simulator can replicate real user conditions. Software needs to be tested on real devices so that they can work in real-world circumstances such as a low battery, incoming calls, weak network strength, and so on. If an in-house lab is not accessible, opt for a cloud-based testing option that offers real devices.

BrowserStack’s cloud Selenium grid offers 2000+ real devices and browsers for automated testing. That means users can run tests on multiple real devices and browsers by simply signing up, logging in, and selecting the required combinations. Testers can also conduct Cypress testing on 30+ real browser versions across Windows and macOS. Detect bugs before users do by testing software in real user conditions with BrowserStack.

Try BrowserStack for Free