Things Every Product Manager Must Know About Testing

Being responsible for developing and maintaining a rapidly evolving tech product is arguably one of the most technical and fast-paced jobs out there. The whole landscape of relevant technologies is also evolving outside your company, and the product adds another layer of complexity to the mix.

From new devices, new features on your favorite cloud platform, the latest APP frameworks, new DB paradigms, and the best IDE for your team, a product manager needs to have strong decision-making skills for ensuring a smooth functioning application.

Despite variabilities like business and programming paradigms, application testing as a complement to application development has been widely adopted as an effective and indispensable tool in the software development life cycle. Let’s try to cover all the basics briefly:  

Every Product Manager must know about Testing

Establishing a healthy testing infrastructure, which is running in sync with the design and development part of the application development effort, is crucial. It is a scientific way of capturing, documenting, sharing, and resolving problems in a software application in a collaborative manner. It helps prevent issues from causing business loss or poor user experiences.

For example, in 2019, British Airways had to cancel 100 and delay 200 flights due to their online portal issues, confusing Heathrow airport. Businesses want to avoid such tragedies, and software testing is the most effective way to ensure such events don’t occur.

Visualizing the testing workflow

Software testing, when done correctly, ensures quality and prevents business failure and customer unhappiness.

1. Choosing The Right Testing Strategy

How and what to test during the SDLC depends mainly on the particulars of the business use case. Developing the most efficient automated testing pyramid is not a trivial task. It is easy to get lost in the sea of online advice and waste time, but you can prevent that if you understand the fundamentals.

Types of Testing

Testing is a broad field of study, and there are many different ways to classify types, but these are the basic layers at which tests are usually implemented:

1. Unit Testing: Testing parts of the codebase through test code, usually part of the same bundle as the working application code.
2. Integration Testing involves testing sections of the codebase as interdependent functions or modules through test code or other tools.
3. System Testing: Testing the workings of an application at the level of features like login, signup, and other supported flows, which validates parts of the application working together.
4. Acceptance Testing: This is usually the final stage of testing in which the fully assembled application with data is tested in a live or pre-production environment. This involves testing with actual or mock users.
5. Performance Testing: With an increase in users, it becomes vital to ensure that the servers can handle the request loads at peak usage times. Also, maintaining end-to-end security at each point of contact between the app and user.

Image Source

Modes of Testing

Based on who is testing:

1. Manual Testing: Testing of any component or aspect of a software application carried out by a human.
2. Automated Testing: Testing of software components carried out by other applications and tools, with minimum human intervention.

Based on the relation between tester and application:

1. Black-box testing: When the tester evaluates an application from outside while not being aware of the internal structure and functions.
2. White-box testing: When testing is carried out with know-how and consideration for the application’s internal structure.

2. Testing DevOps Process

Development Operations or DevOps is a widely used term that refers to a range of activities, including the coding, testing, building, and deployment of software applications. For a product team, the main task is to implement a unit and integration testing plan as part of the technical architecture. Depending on the level of test automation, teams use:

• Continuous Integration (CI): This term is used to describe workflows, where the process of validating code commits of individual contributors and merging with the parent repository is carried out in a seamless automated manner.
• Continuous Delivery/Deployment (CD): This is when a code commit is validated, merged with the parent, and deployed to a testing or production server in the same flow.

Testing in a CI/CD workflow usually involves:

• Writing unit tests: This is a default part of all major programming language packages. Read more about unit testing in Javascript, Python, Java/Kotlin, and Go. With the advent of no-code/low-code and AI-based tools like Devmate, Ponicode, etc, you can also automate parts of the Unit test writing process.
• Testing source code and build: This involves running unit tests in batches, with other tests on the code to test functionality and code quality. After coding, the next task is to bundle the application into a deployable format as specified by the technical architecture. It involves post-processing the source code and installing packages and dependencies as needed. All major cloud hosts like Github, Gitlab, Bitbucket have native CI/CD support.
• Testing backend and database:  For APIs and backend applications like micro-services, integration testing is a very crucial step. Because of complicated architecture and many dependencies, it is vital to ensure proper documentation, performance, and security of REST APIs. Also, the data being passed along needs to be tested to ensure proper values, constraints, etc. Testing DB schema, tables, triggers, etc., separately through specially designed queries and tools is sometimes referred to as Database Testing.
• Testing UI: Testing the application UI in depth is the job of QA teams, but most developers will have a test deployment setup on their dev machine for a brief round of validations that can be performed before code commits. The main goal is to ensure proper integration between the database, APIs, dependencies, and the user interface.

UI tests are best executed on real browsers – devices – OS combinations. Obviously, testing in real user conditions allows the tester to analyze the website or app behavior used by actual customers. If they do not have access to an in-house device lab, they can simply use BrowserStack’s Real Device Cloud.

Try Testing on Real Device Cloud for Free

3. Testing Business With Users

Once an application is deployed, the QA team starts work on the system and acceptance level testing. The starting point for this activity is the test plan document, which details the expected behavior of the application during various business scenarios.

It highlights all the mission-critical workflows and helps the testing team understand the business deeply. With a test plan at hand, the steps are as follows:

Step 1 – Document test cases: For a manual scenario, test cases are maintained in a collaborative documentation platform. The tester is responsible for the execution and reporting of the test runs. But in an automation scenario, the test cases are encoded into a script or 3rd party automation tool like Selenium.

Step 2 – Prepare mock data: Most test scenarios require users to input information. This behavior is replicated during system testing by preparing and using mock data as a substitute for user data. Tools like Visual Studio, Devart, DTM, etc. can help you generate mock data.

Step 3 – Test execution: With the right script and data, the test run takes place. A report is generated at the end, which captures various metrics, helping gauge the outcome of the run. Another priority at this step is to execute on an applicable range of devices and browsers to ensure a wide range of compatibility and accessibility. BrowserStack Live lets you test your local app on 3000+ devices and browsers. You will be spoilt for choice with the range. 

BrowserStack Automate dashboard helps you track and manage automated test runs. Much recently, BrowserStack has introduced support for Selenium 4 tests on Automate allowing users to write tests with a fresh perspective, by tapping into the back end of the browser.

Step 4- Documenting issues and exceptions: During and after test execution, all unexpected behavior is captured and documented in the tracking tool, where it is marked up with meta-data like date/time, location, screenshots/videos, steps to recreate, severity, ownership, etc. Make sure to add video recording to Appium tests on BrowserStack.

Step 5 – Iterate until fixed: The dev team fixes the issues logged in the bug tracker, and the cycle repeats until the QA leads give the green light to the app.

4. Roles and Perspectives of Different Stakeholders

1. Business Leadership: At the level of business leadership, the expectation from the testing process is to ensure that the application(s) is deployed and available to the users while ensuring the best performance, accessibility, and experience. All the documented business requirements and design standards need to be implemented without any discrepancies to fulfill the business goals i.e. core workflows like sign up, log in, booking, product purchase, etc., should work in users’ hands without any hiccups.
2. Technical Leadership: From a technical leadership perspective, including product managers, the main goal is to achieve continuous development and integration flow (CI/CD). Continuous integration with Agile has been widely accepted as the leading Product Management methodology in today’s industry. The technical leaders will maintain a healthy QAOPS workflow, ensuring that the scripts run as scheduled and generate relevant reports periodically. They are constantly monitoring these reports and keeping a keen eye to ensure that the whole application ecosystem delivers on business goals.

At the level of a programmer, the goal of testing is to check and validate any new additions or changes to the code base, before and after they are deployed. In other words, testing for ‘bugs’ in the application code in one’s development, testing/staging, and finally, the production environment. Guidelines for testing within the scope of a particular module are shared with the team, and each contributing member has to write unit tests to cover any new functions or changes they have added to the codebase.

After the initial sanity check on the development side, the QA/QC team takes on testing it further. The testing process can be manual, automated, or mixed. Testers will document application behavior and anomalies in the form of defects logged in a reporting portal, where they can be assigned for fixing and tracking.

After QA approves a version of the app, it is released to a controlled set of users for the first stages of acceptance testing.

5. Reporting and Communications

Once you have a sound strategy, the right tools, and a team in place,  you need dashboards, emails, and messages flowing between the stakeholders, acting as a stream of relevant information regarding tests, bug fix status, performance, etc. It goes like this:

• The business team evaluates the current state of application performance, superimposes near future business goals, and communicates the vision to the technical leadership and design team in the form of business requirement documents.
• Technical leadership with the QA team updates the test plan to include the overall approach collaboratively, involving all stakeholders.
• Updating the test cases in the bug tracker, like a traceability matrix, assigning ownership, and passing the tasks to relevant developers.
• Updating manual and automation scripts to include new cases and doing mock runs. Incase, one is looking to migrate from Manual to Automation Testing, read this.
• Reports that gather and display the status and outputs of all the tests run in a time frame, coming out as dashboards, emails, messages, and notifications.
• It is important to have the relevant information reach all parties to ensure that the right action can be taken in the event of a discrepancy. When an application is being worked on regularly, it helps the stakeholders gauge the application’s overall health.

In Conclusion, 

It is imperative to set up a system where all participants can effectively collaborate on the testing process. To get the best results, you need the whole team to work together.  Armed with the right knowledge and tools, you can effectively improve the quality of deliverables and achieve great things with your product. Although some would argue about the resource-heavy and time-consuming nature of the testing workflow, in a proper development environment, testing provides a unique opportunity to improve the quality of any software application without having to risk any real user or business opportunities.

In an evolving ecosystem of technology, BrowserStack offers an industry-leading testing infrastructure to maximize the productivity of your software testing team and ship quality releases at the speed of Agile. Get onboarded to streamline your product management. 

Get Access To BrowserStack Enterprise