Mastering Test Automation with ChatGPT

Test automation is essential for bug-free software and ensuring desired user experience. As technology advances, so does the complexity of test automation. 

Keeping up with the latest tools and techniques can be a daunting task, especially for those who are not familiar with coding or are new to the field. 

However, modern AI technology is crossing the horizon, entering the mainstream with powerful tools like ChatGPT, which can, among thousand other things, talk to you and teach you how to read and write code like a pro-human.

In this article, let us go through the process of mastering test automation using BrowserStack hybrid cloud and OpenAI GPT technology.

What is ChatGPT?

ChatGPT is a state-of-the-art language model developed by OpenAI. By using natural language processing-based artificial intelligence, it allows for the automation of tasks that involve human-like interactions, such as an AI chatbot or virtual assistant.

OpenAI intends to use different interfaces like Playground, ChatGPT, and APIs to make its AI capabilities more accessible and user-friendly for a wide range of users and applications.

The playground interface allows users to experiment with and interact with OpenAI’s AI models visually and intuitively. This interface is designed to be easy to use and provides finer controls over model dynamics and I/O.

ChatGPT is a web application based on an AI model that allows users to interact with OpenAI’s AI naturally and conversationally. It can understand and respond to natural language input and can keep track of previous conversations to provide more accurate and relevant answers.

ChatGPT can generate text in various formats, such as structured data, code snippets, and annotations, which can be used to automate tasks such as code generation and testing.

Here are a few ways in which ChatGPT can be used to learn how to code:

1. Code generation: it can generate code snippets based on natural language prompts, providing a great way for beginners to learn the basic syntax and structure of a programming language.
2. Code completion: it can be used to complete partially written code, providing guidance on the next steps and helping beginners understand how to solve a particular problem.
3. Code explanation: it can be used to explain the purpose and function of specific lines of code, helping beginners understand how different code components work together.
4. Debugging: it can be used to identify and explain errors in code, helping beginners learn how to debug and troubleshoot their own code.
5. Project guidance: it can provide guidance and suggestions for project structure, best practices, and libraries to use for beginners who want to build a project.

Overall, it is a positive development, and the vast usage demonstrates the growing interest among users to have high-capability AI tools as part of their arsenal.

That said, please note that ChatGPT is a powerful language model, but it is not a replacement for human teachers, mentors, or experienced developers. It can be a great aid in learning to code, but it should be used in conjunction with other resources and guidance from experienced developers.

What is BrowserStack Automate?

BrowserStack Automate is a testing platform that allows users to run automated tests on a variety of devices and browsers. With a wide range of configurations and capabilities, it is a great tool for testing web and mobile applications. 

One of the biggest advantages of using BrowserStack Automate is its cloud platform, which allows for easy access to a wide range of devices and browsers for testing. 

This eliminates the need for a physical device lab and allows for testing on a variety of configurations, ensuring that your application works seamlessly on different devices and browsers.

Another advantage of BrowserStack is its integration with Selenium, Appium, and other popular test automation frameworks. This integration allows for the automation of web and mobile applications and provides a wide range of capabilities to test your application effectively.

Try BrowserStack Automate

How to use ChatGPT and Automate for Test Automation?

Combining the power of BrowserStack Automate and ChatGPT can bring significant enhancements to your test automation process. 

Before getting started, you will need to set up an account with BrowserStack.

And you will need OpenAI account to access ChatGPT.

ChatGPT can be used in various ways when trying to learn or implement test automation for any kind of scenario, including but not limited to:

1. Unit Testing: For front-end projects built on several languages such as Javascript, Python, Java, Swift, etc. ChatGPT can be used to generate helpful syntax references for writing unit tests properly.
2. Integration Testing: ChatGPT can be used to generate test scripts for integration testing with the help of a framework like Selenium. This can be used for testing front-end web applications built with Javascript, HTML, and CSS.
3. API Testing: ChatGPT can be used to generate test scripts for API testing with the help of a framework like Rest Assured or Postman. This can be used for testing backend APIs built with languages such as Java, Python, NodeJS, etc.
4. Mobile Testing: ChatGPT can be used to generate test scripts for mobile testing with the help of a framework like Appium. This can be used for testing mobile applications built with languages such as Java, Swift, etc.
5. End To End Testing: ChatGPT can be used to generate test scripts for end-to-end testing with the help of a framework like Selenium, Puppeteer, etc. This can be used for testing end-to-end scenarios for web applications built with Javascript, HTML, and CSS.

ChatGPT works in threads of conversation much like a messaging app and maintains context for future reference in the chat history.

Examples of Using ChatGPT for Automation Testing

Test planning with ChatGPT

Example 1: Suppose you had a website bstackdemo.com on which you have built some functionality, and you want to plan the test automation. 

You can use GPT in such cases by giving it the details on functional components and their use and asking it to generate a test plan.

Prompt: “Bstackdemo.com is an eCommerce website. Prepare a test plan for automation tests to carry out the following validations on the website:

• Open browserstack.com. Success if page loads with title ‘StackDemo’
• Under vendors, select ‘Google’. Success if product list is filtered.
• Add pixel 4 to cart. Success if overlay with title ‘Bag’ shows entry.
• Click on checkout. Success if page redirects to ‘/signin?checkout=true’”

ChatGPT:

And just like that, you have enough template material to start a test document. You can share this with your team and start building on this basic plan.

Notice not only did It understand the task from limited information, but it also figured out an appropriate platform and automation tool to test with. Once your plan is ready you can test it on BrowserStack.

Example 2: You can use ChatGPT to reformat and prepare any data for any other format. 

Let’s say you want the test plan in a format where it could be easy for you to enter it into a Bug tracking platform like Jira or Github. 

You could use ChatGPT in such case by prompting further in the same thread :

Prompt: “Prepare test case table to enter into Jira tickets ”

ChatGPT:

Unit testing with ChatGPT

Example 1: You can give ChatGPT a piece of code and it will write unit tests for it in the desired format.

Prompt: “Write unit tests to test the following code:

class SomePlugin extends React.Component {
componentDidMount() {
this.$el = $(this.el);
this.$el.somePlugin();
}
componentWillUnmount() {
this.$el.somePlugin('destroy');
}
render() {
return <div ref={el => this.el = el} />;
}
}

ChatGPT: 

Not only can it recognize the language of input code without any information from our side, but it also uses a correct library called Enzyme, and goes on to explain the examples in further depth with appropriate disclaimers.

Example 2: If you are a beginner and just want to learn concepts, you can ask ChatGPT to give examples and explain how unit testing works in any popular language or framework.

Prompt: “How to write unit tests for iOS using XCtest? Explain with an example.”

ChatGPT: Responds with definitions, examples, and a detailed explanation:

The power to seamlessly present concepts, code samples, and explanations for code is what makes ChatGPT a powerful tool to be deployed in various coding scenarios including test automation.

Write test automation in any framework with ChatGPT

Example 1: Web testing with Selenium

Continuing with the test plans generated earlier, you can use ChatGPT to now write test automation code for the same.

Prompt: “Write a BrowserStack Automate script in nodeJS with Selenium to 

1. Open the Bstackdemo.com website in the Chrome browser. 
2. Verify that the title of the webpage is “StackDemo”. 
3. Verify that the homepage loads correctly with all elements displayed as expected.”

ChatGPT:

Example 2: Mobile testing with Appium

Prompt: “How to write automation test in BrowserStack using Appium? Explain with a short example.”

ChatGPT:

Example 3: Web crawl using Puppeteer

Prompt: “Write a puppeteer script to open browserstack.com/guide and extract the first 3 articles from the page.”

ChatGPT:

Just like this, by varying the prompts appropriately, you can generate code samples for the programming language and automation framework of your choice.

These code snippets are never 100% on syntax, but enough for you to have a coarse structure that can be further worked upon and refined with the help of official guides and, again, ChatGPT.

Advanced Script level operations with ChatGPT

Example 1: You can use the AI to explain to you a part of the code or the entire file; you can just copy and paste and ask it to explain it back by:

Prompt: Explain the following code:

const webdriver = require('selenium-webdriver');
async function runTestWithCaps (capabilities) {
let driver = new webdriver.Builder()
.usingServer('http://<YOUR-SERVER>.browserstack.com/wd/hub')
.withCapabilities({
...capabilities,
...capabilities['browser'] && { browserName: capabilities['browser']} // Because NodeJS language binding requires browserName to be defined
})
.build();
try {
await driver.get("https://bstackdemo.com/");
await driver.wait(webdriver.until.titleMatches(/StackDemo/i), 10000);
// locating product on webpage and getting name of the product
let productText = await driver
.findElement(webdriver.By.xpath('//*[@id="1"]/p'))
.getText();
// clicking the 'Add to cart' button
await driver.findElement(webdriver.By.xpath('//*[@id="1"]/div[4]')).click();
// waiting until the Cart pane has been displayed on the webpage
driver.findElement(webdriver.By.className("float-cart__content"));
// locating product in cart and getting name of the product in cart
let productCartText = await driver
.findElement(
webdriver.By.xpath(
'//*[@id="__next"]/div/div/div[2]/div[2]/div[2]/div/div[3]/p[1]'
)
)
.getText();
// checking whether product has been added to cart by comparing product name
if(productCartText !== productText)
throw new Error("");
await driver.executeScript(
'browserstack_executor: {"action": "setSessionStatus", "arguments": {"status":"passed","reason": "Product has been successfully added to the cart!"}}'
);
} catch (e) {
await driver.executeScript(
'browserstack_executor: {"action": "setSessionStatus", "arguments": {"status":"failed","reason": "Some elements failed to load!"}}'
);
}
await driver.quit();
}
const capabilities1 = {
'bstack:options' : {
"deviceName": "iPhone 14 Pro Max",
"osVersion": "16",
"deviceOrientation": "portrait",
"buildName" : "browserstack-build-1",
"sessionName" : "Parallel test 1",
},
"browserName": "ios"
}
const capabilities2 = {
'bstack:options' : {
"deviceName": "Google Pixel 7 Pro",
"osVersion": "13.0",
"deviceOrientation": "portrait",
"buildName" : "browserstack-build-1",
"sessionName" : "Parallel test 2",
},
"browserName": "android"
}
const capabilities3 = {
'bstack:options' : {
"os": "Windows",
"osVersion": "11",
"browserVersion": "latest-beta",
"buildName" : "browserstack-build-1",
"sessionName" : "Parallel test 3",
},
"browserName": "Edge"
}
runTestWithCaps(capabilities1);
runTestWithCaps(capabilities2);
runTestWithCaps(capabilities3);

ChatGPT: 

Pretty good summary by ChatGPT.

Example 2: Now, if you want to change something in the code – like instead of running 3 tests, just run on one device and change test parameters, etc. You can make ChatGPT do that by prompting something like this:

Prompt: “Change the provided example code, and instead of running 3 tests in parallel, run only the first one on the iPhone and remove others. After that, change the part where it matches the title of the page and product name and remove it. In its place, after opening bstackdemo.com, just check and log the first h1 element on the page.”

ChatGPT: 

It seamlessly understands sandwiched commands and outputs this code, which when we ran through diff checker against the original code we get this:

This is not far from what a human developer’s git revision before code push would look like. 

As you can see, it can maintain this flow of back and forth irrespective of how long the conversation is and make the right changes in the right place.

Ideally, you should start with small bits and try to put them together to get the most out of its capabilities. Feeding it large prompts could lead to difficulties while trying to control it and also a bad server-side response time.

Data analysis and summarisation

ChatGPT can be used by developers and QA professionals to improve their routine reporting and data analysis activities in several ways, especially related to testing automation report analysis.

ChatGPT can be used to summarize the test execution results, providing an overview of the test results, including the number of tests passed, failed, and skipped.

Example 1: Understand XML report from Cypress Junit

Prompt: “Please explain this Cypress Junit report: 

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="Cypress Test Suite" tests="3" failures="0" errors="0" skipped="0" timestamp="2022-05-12T12:00:00Z" time="12.345">
<testcase classname="Homepage Test" name="Verify homepage loads correctly" time="3.456"></testcase>
<testcase classname="Homepage Test" name="Verify title is correct" time="2.345"></testcase>
<testcase classname="Homepage Test" name="Verify navigation links" time="6.789" failures="1">
<failure message="Navigation link 'Contact Us' not found">
CypressError: Timed out retrying: expected to find element: '.nav-link[href="/contact"]', but never found it.
</failure>
</testcase>
</testsuite>
</testsuites>

ChatGPT:

Example 2: Make sense of system/coding errors with ChatGPT.

Prompt: “What could be the cause of this error in Xcode:

*** Terminating app due to uncaught exception ‘NSUnknownKeyException’, reason: ‘[<UIViewController 0x6e36ae0> setValue:forUndefinedKey:]: this class is not key value coding-compliant for the key XXX.’

ChatGPT: 

Example 3: Sharpen programming language skills with ChatGPT.

Prompt: “Why is processing a sorted array faster than processing an unsorted array in C++?”

ChatGPT:

ChatGPT as an all-purpose AI Assistant

Now, for the last section of our series of examples, let us look at the most general and powerful capability of ChatGPT, which is to behave like a real human with thoughts and opinions on various aspects of development, automation testing or whatever else you want to discuss with it.

Example 1:  ChatGPT for help with technical decisions.

Prompt: “We want to build a 5-page web app as a prototype for our new product series catalog, we have about 15 days to do that before the meeting and we have not decided what frameworks to use, please suggest the best frameworks/libraries for TS, CSS, DB, build, CICD and automation testing.”

ChatGPT:

Example 2: ChatGPT for imaginative tasks

Prompt: “How do you foresee the Cloud test automation market evolving in the next 5 years to come?”

ChatGPT:

Limitations of ChatGPT

ChatGPT has certain limitations and is constantly changing and evolving with more widespread adoption.

Here is a quick summary of its limitations:

1. Lack of domain knowledge: While ChatGPT can generate text that is similar to human writing, it may lack the specific domain knowledge or expertise that a human expert would have in certain areas.
2. Factual errors: Since ChatGPT is trained on a vast amount of text data, it may inadvertently include factual errors or misinformation present in the training data.
3. Code errors: Code generated by ChatGPT may not always work as it may not fully understand the specific context or requirements of the task, and may contain errors or be less optimized. It’s important to thoroughly verify and test the generated code before using it in any project.
4. Biases: Since the model is trained on a large amount of text data from the internet, it may have learned biases present in that data. This can lead to the generation of offensive or discriminatory content.
5. Limited accountability: Since ChatGPT is a machine learning model, it can be hard to understand how it generates specific outputs, making it difficult to hold the model accountable for any inaccuracies or errors.
6. Limited context understanding: ChatGPT is trained to generate text based on the input it receives, but it may not fully understand the context or intent of that input. This can lead to confusion or misinterpretation of the input, and the output may not be entirely accurate.

OpenAI, the company behind ChatGPT, has been working on ways to mitigate these issues., it is essential always to verify the information generated by GPT models, especially when used for critical tasks.

Since this is a new development, the functionality provided by ChatGPT is not limited to test automation. 

Think of it as a sum of all tech knowledge on the internet before 2021 made accessible through a highly functional chatbot that can process and present information from different perspectives in different contexts.

Summary

Mastering test automation with ChatGPT and BrowserStack Automate is a powerful way for any organization looking to improve the efficiency and effectiveness of their testing processes. 

With ChatGPT, teams can easily plan, execute and analyze their tests, while BrowserStack Automate allows them to run those tests across a wide range of browsers and devices. 

Together, these tools provide a comprehensive solution for test automation that is easy to use and provides fine control over test dynamics. 

By leveraging the power of AI, teams can automate repetitive tasks and access deeper insights into their test data, making it easier to identify and diagnose issues, predict and prevent defects and improve overall quality. 

While there are limitations to using GPT for test automation, with the right approach and the right use cases, it can be a powerful tool that can help teams to improve their testing process and ultimately to deliver better products.