Mastering Test Automation with ChatGPT

Artificial Intelligence (AI) is revolutionizing software testing by automating repetitive tasks, enhancing accuracy, and speeding up test execution.

ChatGPT, as an AI-powered assistant, takes test automation to the next level by helping QA teams write, debug, and optimize test scripts across frameworks.

This article describes how to use ChatGPT for test automation, its benefits and limitations.

What is ChatGPT?

ChatGPT is a state-of-the-art language model developed by OpenAI. By using natural language processing-based artificial intelligence, it allows for the automation of tasks that involve human-like interactions, such as an AI chatbot or virtual assistant.

OpenAI intends to use different interfaces like Playground, ChatGPT, and APIs to make its AI capabilities more accessible and user-friendly for a wide range of users and applications.

The playground interface allows users to experiment with and interact with OpenAI’s AI models visually and intuitively. This interface is designed to be easy to use and provides finer controls over model dynamics and I/O.

ChatGPT is a web application based on an AI model that allows users to interact with OpenAI’s AI naturally and conversationally. It can understand and respond to natural language input and can keep track of previous conversations to provide more accurate and relevant answers.

ChatGPT can generate text in various formats, such as structured data, code snippets, and annotations, which can be used to automate tasks such as code generation and testing.

Here are a few ways in which ChatGPT can be used to learn how to code:

1. Code generation: it can generate code snippets based on natural language prompts, providing a great way for beginners to learn the basic syntax and structure of a programming language.
2. Code completion: it can be used to complete partially written code, providing guidance on the next steps and helping beginners understand how to solve a particular problem.
3. Code explanation: it can be used to explain the purpose and function of specific lines of code, helping beginners understand how different code components work together.
4. Debugging: it can be used to identify and explain errors in code, helping beginners learn how to debug and troubleshoot their own code.
5. Project guidance: it can provide guidance and suggestions for project structure, best practices, and libraries to use for beginners who want to build a project.

Overall, it is a positive development, and the vast usage demonstrates the growing interest among users to have high-capability AI tools as part of their arsenal.

That said, please note that ChatGPT is a powerful language model, but it is not a replacement for human teachers, mentors, or experienced developers. It can be a great aid in learning to code, but it should be used in conjunction with other resources and guidance from experienced developers.

Benefits of using ChatGPT for Test Automation

Test automation has evolved significantly with AI-driven tools like ChatGPT. It makes it faster, smarter, and more efficient. Here’s how ChatGPT can enhance your test automation process:

1. Faster Test Script Generation

Writing test scripts can be time-consuming, but ChatGPT can generate Selenium, Appium, or Playwright scripts quickly. Whether you need UI, API, or database tests, it can provide a solid starting point, reducing development time.

2. Better Debugging & Troubleshooting

If you are stuck on an error, ChatGPT can analyze stack traces, suggest fixes, and help resolve issues in your automation framework. It acts as a real-time assistant for debugging test failures.

3. Enhanced Test Coverage

AI-powered suggestions ensure you don’t miss edge cases. ChatGPT can help you think through various test scenarios, covering both functional and non-functional aspects like performance and security testing.

4. Efficient Test Data Generation

Creating diverse and meaningful test data is crucial. ChatGPT can generate randomized datasets, mock API responses, or structured data to improve test reliability and coverage.

5. Improved Collaboration with Non-Technical Teams

ChatGPT simplifies complex automation concepts, making it easier for QA teams to collaborate with business analysts, product managers, and developers. It can even convert plain English requirements into automated test cases.

6. Faster Learning Curve for Testers

Are you new to test automation? ChatGPT can act as an on-demand mentor, explaining concepts and best practices and helping you learn automation frameworks like Selenium, Cypress, and Playwright.

7. Integration with CI/CD Pipelines

Need help setting up test automation in CI/CD pipelines? ChatGPT can guide you on integrating tests into Jenkins, GitHub Actions, or GitLab CI/CD, ensuring smooth deployments.

8. Smoother Maintenance of Test Scripts

As applications evolve, maintaining test scripts is crucial. ChatGPT can refactor outdated scripts, suggest improvements, and help with migration to new automation frameworks.

How to use ChatGPT and Automate for Test Automation?

Combining the power of BrowserStack Automate and ChatGPT can bring significant enhancements to your test automation process.

Before getting started, you will need to set up an account with BrowserStack.

And you will need OpenAI account to access ChatGPT.

ChatGPT can be used in various ways when trying to learn or implement test automation for any kind of scenario, including but not limited to:

1. Unit Testing: For front-end projects built on several languages such as Javascript, Python, Java, Swift, etc. ChatGPT can be used to generate helpful syntax references for writing unit tests properly.
2. Integration Testing: ChatGPT can be used to generate test scripts for integration testing with the help of a framework like Selenium. This can be used for testing front-end web applications built with Javascript, HTML, and CSS.
3. API Testing: ChatGPT can be used to generate test scripts for API testing with the help of a framework like Rest Assured or Postman. This can be used for testing backend APIs built with languages such as Java, Python, NodeJS, etc.
4. Mobile Testing: ChatGPT can be used to generate test scripts for mobile testing with the help of a framework like Appium. This can be used for testing mobile applications built with languages such as Java, Swift, etc.
5. End To End Testing: ChatGPT can be used to generate test scripts for end-to-end testing with the help of a framework like Selenium, Puppeteer, etc. This can be used for testing end-to-end scenarios for web applications built with Javascript, HTML, and CSS.

ChatGPT works in threads of conversation much like a messaging app and maintains context for future reference in the chat history.

Example Use Cases of using ChatGPT for Automation Testing

Here are the example use cases of how to use ChatGPT for test automation:

Test planning with ChatGPT

Example 1: Suppose you had a website bstackdemo.com on which you have built some functionality, and you want to plan the test automation.

You can use GPT in such cases by giving it the details on functional components and their use and asking it to generate a test plan.

Prompt: “Bstackdemo.com is an eCommerce website. Prepare a test plan for automation tests to carry out the following validations on the website:

• Open browserstack.com. Success if page loads with title ‘StackDemo’
• Under vendors, select ‘Google’. Success if product list is filtered.
• Add pixel 4 to cart. Success if overlay with title ‘Bag’ shows entry.
• Click on checkout. Success if page redirects to ‘/signin?checkout=true’”

ChatGPT:

And just like that, you have enough template material to start a test document. You can share this with your team and start building on this basic plan.

Notice not only did It understand the task from limited information, but it also figured out an appropriate platform and automation tool to test with. Once your plan is ready you can test it on BrowserStack.

Example 2: You can use ChatGPT to reformat and prepare any data for any other format.

Let’s say you want the test plan in a format where it could be easy for you to enter it into a Bug tracking platform like Jira or Github.

You could use ChatGPT in such case by prompting further in the same thread :

Prompt: “Prepare test case table to enter into Jira tickets ”

ChatGPT:

Write sample Test Cases with ChatGPT

ChatGPT can help developers and QA professionals create well-structured, clear, and comprehensive test cases. Whether you’re defining test scenarios for UI, API, or performance testing, ChatGPT can speed up the process while ensuring coverage and consistency.

It can generate test cases based on user inputs, acceptance criteria, or functional requirements, reducing manual effort and improving collaboration between technical and non-technical stakeholders.

Example 1: Generate a Test Case for a Login Functionality

Prompt:

“Write a detailed test case for verifying login functionality with valid and invalid credentials.”

ChatGPT Response:

Unit testing with ChatGPT

Example 1: You can give ChatGPT a piece of code and it will write unit tests for it in the desired format.

Prompt: “Write unit tests to test the following code:

class SomePlugin extends React.Component {
componentDidMount() {
this.$el = $(this.el);
this.$el.somePlugin();
}
componentWillUnmount() {
this.$el.somePlugin('destroy');
}
render() {
return <div ref={el => this.el = el} />;
}
}

ChatGPT:

Not only can it recognize the language of input code without any information from our side, but it also uses a correct library called Enzyme, and goes on to explain the examples in further depth with appropriate disclaimers.

Example 2: If you are a beginner and just want to learn concepts, you can ask ChatGPT to give examples and explain how unit testing works in any popular language or framework.

Prompt: “How to write unit tests for iOS using XCtest? Explain with an example.”

ChatGPT: Responds with definitions, examples, and a detailed explanation:

The power to seamlessly present concepts, code samples, and explanations for code is what makes ChatGPT a powerful tool to be deployed in various coding scenarios including test automation.

Write test automation in any framework with ChatGPT

Example 1: Web testing with Selenium

Continuing with the test plans generated earlier, you can use ChatGPT to now write test automation code for the same.

Prompt: “Write a BrowserStack Automate script in nodeJS with Selenium to

1. Open the Bstackdemo.com website in the Chrome browser.
2. Verify that the title of the webpage is “StackDemo”.
3. Verify that the homepage loads correctly with all elements displayed as expected.”

ChatGPT:

Example 2: Mobile testing with Appium

Prompt: “How to write automation test in BrowserStack using Appium? Explain with a short example.”

ChatGPT:

Example 3: Web crawl using Puppeteer

Prompt: “Write a puppeteer script to open browserstack.com/guide and extract the first 3 articles from the page.”

ChatGPT:

Just like this, by varying the prompts appropriately, you can generate code samples for the programming language and automation framework of your choice.

These code snippets are never 100% on syntax, but enough for you to have a coarse structure that can be further worked upon and refined with the help of official guides and, again, ChatGPT.

Advanced Script level operations with ChatGPT

Example 1: You can use the AI to explain to you a part of the code or the entire file; you can just copy and paste and ask it to explain it back by:

Prompt: Explain the following code:

const webdriver = require('selenium-webdriver');
async function runTestWithCaps (capabilities) {
let driver = new webdriver.Builder()
.usingServer('http://<YOUR-SERVER>.browserstack.com/wd/hub')
.withCapabilities({
...capabilities,
...capabilities['browser'] && { browserName: capabilities['browser']} // Because NodeJS language binding requires browserName to be defined
})
.build();
try {
await driver.get("https://bstackdemo.com/");
await driver.wait(webdriver.until.titleMatches(/StackDemo/i), 10000);
// locating product on webpage and getting name of the product
let productText = await driver
.findElement(webdriver.By.xpath('//*[@id="1"]/p'))
.getText();
// clicking the 'Add to cart' button
await driver.findElement(webdriver.By.xpath('//*[@id="1"]/div[4]')).click();
// waiting until the Cart pane has been displayed on the webpage
driver.findElement(webdriver.By.className("float-cart__content"));
// locating product in cart and getting name of the product in cart
let productCartText = await driver
.findElement(
webdriver.By.xpath(
'//*[@id="__next"]/div/div/div[2]/div[2]/div[2]/div/div[3]/p[1]'
)
)
.getText();
// checking whether product has been added to cart by comparing product name
if(productCartText !== productText)
throw new Error("");
await driver.executeScript(
'browserstack_executor: {"action": "setSessionStatus", "arguments": {"status":"passed","reason": "Product has been successfully added to the cart!"}}'
);
} catch (e) {
await driver.executeScript(
'browserstack_executor: {"action": "setSessionStatus", "arguments": {"status":"failed","reason": "Some elements failed to load!"}}'
);
}
await driver.quit();
}
const capabilities1 = {
'bstack:options' : {
"deviceName": "iPhone 14 Pro Max",
"osVersion": "16",
"deviceOrientation": "portrait",
"buildName" : "browserstack-build-1",
"sessionName" : "Parallel test 1",
},
"browserName": "ios"
}
const capabilities2 = {
'bstack:options' : {
"deviceName": "Google Pixel 7 Pro",
"osVersion": "13.0",
"deviceOrientation": "portrait",
"buildName" : "browserstack-build-1",
"sessionName" : "Parallel test 2",
},
"browserName": "android"
}
const capabilities3 = {
'bstack:options' : {
"os": "Windows",
"osVersion": "11",
"browserVersion": "latest-beta",
"buildName" : "browserstack-build-1",
"sessionName" : "Parallel test 3",
},
"browserName": "Edge"
}
runTestWithCaps(capabilities1);
runTestWithCaps(capabilities2);
runTestWithCaps(capabilities3);

ChatGPT:

Pretty good summary by ChatGPT.

Example 2: Now, if you want to change something in the code – like instead of running 3 tests, just run on one device and change test parameters, etc. You can make ChatGPT do that by prompting something like this:

Prompt: “Change the provided example code, and instead of running 3 tests in parallel, run only the first one on the iPhone and remove others. After that, change the part where it matches the title of the page and product name and remove it. In its place, after opening bstackdemo.com, just check and log the first h1 element on the page.”

ChatGPT:

It seamlessly understands sandwiched commands and outputs this code, which when we ran through diff checker against the original code we get this:

This is not far from what a human developer’s git revision before code push would look like.

As you can see, it can maintain this flow of back and forth irrespective of how long the conversation is and make the right changes in the right place.

Ideally, you should start with small bits and try to put them together to get the most out of its capabilities. Feeding it large prompts could lead to difficulties while trying to control it and also a bad server-side response time.

Create Test Data with ChatGPT

ChatGPT can generate diverse and structured test data for automation testing. Whether you need randomized inputs, mock API responses, or predefined datasets, ChatGPT can generate valid and invalid test data for different scenarios.

Example 1: Generate Random User Data for Login Testing

Prompt:

“Generate a set of 5 valid and 5 invalid email-password combinations for testing a login page.”

ChatGPT Response:

Change and Fix Code with ChatGPT

ChatGPT can assist developers and QA professionals in modifying, optimizing, and fixing test automation scripts. Whether you are debugging Selenium, Cypress, or Playwright tests, it can help identify issues, suggest fixes, and refactor code for better efficiency.

ChatGPT can analyze error messages, optimize locators, update outdated syntax, and even migrate test scripts from one framework to another.

Example 1: Fix a Broken Selenium Test

Prompt:
*“My Selenium test for verifying login is failing. Here’s the code:

python

from selenium import webdriver  

from selenium.webdriver.common.by import By  




driver = webdriver.Chrome()  

driver.get("https://example.com/login")  




username = driver.find_element(By.ID, "user")  

password = driver.find_element(By.ID, "pass")  

login_button = driver.find_element(By.ID, "login-btn")  




username.send_keys("testuser")  

password.send_keys("password123")  

login_button.click()  




assert "Dashboard" in driver.title  

driver.quit()

The test fails with ‘NoSuchElementException’ for the login button. Can you help?”*

ChatGPT Response:

Data Analysis and Summarisation

ChatGPT can be used by developers and QA professionals to improve their routine reporting and data analysis activities in several ways, especially related to testing automation report analysis.

ChatGPT can be used to summarize the test execution results, providing an overview of the test results, including the number of tests passed, failed, and skipped.

Example 1: Understand XML report from Cypress Junit

Prompt: “Please explain this Cypress Junit report:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="Cypress Test Suite" tests="3" failures="0" errors="0" skipped="0" timestamp="2022-05-12T12:00:00Z" time="12.345">
<testcase classname="Homepage Test" name="Verify homepage loads correctly" time="3.456"></testcase>
<testcase classname="Homepage Test" name="Verify title is correct" time="2.345"></testcase>
<testcase classname="Homepage Test" name="Verify navigation links" time="6.789" failures="1">
<failure message="Navigation link 'Contact Us' not found">
CypressError: Timed out retrying: expected to find element: '.nav-link[href="/contact"]', but never found it.
</failure>
</testcase>
</testsuite>
</testsuites>

ChatGPT:

Example 2: Make sense of system/coding errors with ChatGPT.

Prompt: “What could be the cause of this error in Xcode:

*** Terminating app due to uncaught exception ‘NSUnknownKeyException’, reason: ‘[<UIViewController 0x6e36ae0> setValue:forUndefinedKey:]: this class is not key value coding-compliant for the key XXX.’

ChatGPT:

Example 3: Sharpen programming language skills with ChatGPT.

Prompt: “Why is processing a sorted array faster than processing an unsorted array in C++?”

ChatGPT:

ChatGPT as an All-Purpose AI Assistant

Now, for the last section of our series of examples, let us look at the most general and powerful capability of ChatGPT, which is to behave like a real human with thoughts and opinions on various aspects of development, automation testing or whatever else you want to discuss with it.

Example 1:  ChatGPT for help with technical decisions.

Prompt: “We want to build a 5-page web app as a prototype for our new product series catalog, we have about 15 days to do that before the meeting and we have not decided what frameworks to use, please suggest the best frameworks/libraries for TS, CSS, DB, build, CICD and automation testing.”

ChatGPT:

Example 2: ChatGPT for imaginative tasks

Prompt: “How do you foresee the Cloud test automation market evolving in the next 5 years to come?”

ChatGPT:

Limitations of ChatGPT

ChatGPT has certain limitations and is constantly changing and evolving with more widespread adoption.

Here is a quick summary of its limitations:

1. Lack of domain knowledge: While ChatGPT can generate text that is similar to human writing, it may lack the specific domain knowledge or expertise that a human expert would have in certain areas.
2. Factual errors: Since ChatGPT is trained on a vast amount of text data, it may inadvertently include factual errors or misinformation present in the training data.
3. Code errors: Code generated by ChatGPT may not always work as it may not fully understand the specific context or requirements of the task, and may contain errors or be less optimized. It’s important to thoroughly verify and test the generated code before using it in any project.
4. Biases: Since the model is trained on a large amount of text data from the internet, it may have learned biases present in that data. This can lead to the generation of offensive or discriminatory content.
5. Limited accountability: Since ChatGPT is a machine learning model, it can be hard to understand how it generates specific outputs, making it difficult to hold the model accountable for any inaccuracies or errors.
6. Limited context understanding: ChatGPT is trained to generate text based on the input it receives, but it may not fully understand the context or intent of that input. This can lead to confusion or misinterpretation of the input, and the output may not be entirely accurate.

OpenAI, the company behind ChatGPT, has been working on ways to mitigate these issues., it is essential always to verify the information generated by GPT models, especially when used for critical tasks.

Since this is a new development, the functionality provided by ChatGPT is not limited to test automation.

Think of it as a sum of all tech knowledge on the internet before 2021 made accessible through a highly functional chatbot that can process and present information from different perspectives in different contexts.

Best Practices of using ChatGPT for Test Automation

ChatGPT is a powerful tool for accelerating test automation, but to get the best results, it’s essential to use it effectively.

Here are some best practices to maximize the impact on your testing workflow.

1. Provide Clear and Specific Prompts

When asking ChatGPT to generate test scripts, debug errors, or suggest improvements, be as specific as possible. Include details such as the programming language, testing framework, and expected behavior.

Example:

“Generate a Selenium test script in Python to validate a login form with valid and invalid credentials.”

Avoid:

“Write a test case for login.” (Too vague)

2. Validate and Refine Generated Code

While ChatGPT provides a solid starting point, always review and test the generated scripts before using them in production. AI-generated code may contain syntax errors or inefficient logic that needs fine-tuning.

3. Use ChatGPT for Debugging and Troubleshooting

ChatGPT can analyze error messages and suggest fixes. If a test script is failing, paste the error log along with the relevant code snippet, and ChatGPT can help identify and resolve the issue.

Example Prompt:

“My Cypress test is failing with a ‘cy.click() failed because the element is hidden’ error. Here’s my test script. How do I fix it?”

4. Leverage ChatGPT for Test Optimization

ChatGPT can help optimize test scripts for better performance, such as reducing flakiness, improving element locators, and implementing best practices like explicit waits instead of hardcoded sleep statements.

Example Prompt:

“Refactor my Selenium test to use explicit waits instead of time.sleep().”

5. Integrate ChatGPT for Test Case Documentation

ChatGPT can convert requirements into structured test cases, making it easier to maintain documentation and ensure test coverage.

Example Prompt:

“Convert this user story into test cases: ‘As a user, I should be able to reset my password via email verification.”

6. Avoid Over-Reliance

AI is a great assistant but not a replacement for human expertise. Always review, validate, and refine ChatGPT’s output to ensure accuracy and alignment with business requirements.

How to use ChatGPT with BrowserStack for Test Automation?

AI helps you write, debug, and optimize scripts, but real-world execution ensures accuracy, compatibility, and seamless user experience.

Here is how you can combine ChatGPT and BrowserStack Automate for faster test creation and reliable cross browser testing.

Step 1. Generate Test Scripts with ChatGPT

• Ask ChatGPT to create a test script (for example, Selenium, Cypress, Playwright).
• Modify or debug existing scripts using AI suggestions.

Step 2. Set Up BrowserStack Automate

• Sign up for a BrowserStack Automate account.
• Get your username and access key from the dashboard.

Step 3. Modify Your Test Script for BrowserStack

• Add BrowserStack capabilities (for example, browser, OS, version).
• Use Remote WebDriver to run tests on BrowserStack.

Step 4. Execute Tests on Real Devices & Browsers

• Run tests across multiple devices and browsers.
• View real-time logs, screenshots, and video recordings on the BrowserStack dashboard.

Step 5. Debug and Optimize

• Use ChatGPT to fix test failures based on error logs.
• Run tests in parallel on BrowserStack for faster execution.

Conclusion

You can use AI to generate and optimize test cases or scripts, but execution on real devices is essential for accuracy and cross-browser compatibility. While ChatGPT accelerates test creation and debugging, reliable automation testing requires real testing environments.

With BrowserStack Automate, you can run AI-generated tests on 3,500+ real browsers and devices, ensuring seamless functionality and error-free releases.

Try BrowserStack Now