How to set goals for Software Quality

What is Software Quality and Why is it important?

The International Software Testing Qualifications Board (ISTQB), one of the world’s leading software certification bodies defines software quality as “The totality of functionality and features of a software product that bear on its ability to satisfy stated or implied needs”, whereas the IEEE defines software quality as “The degree to which a system, component, or process meets customer or user needs or expectations.”

To get a better idea of what these definitions mean in the context of setting SMART goals for software quality, it is necessary to understand how the lack of Quality has impacted some major organizations in the past, through the following Case Studies.

The Mariner 1 Spacecraft, 1962. NASA – In 1962, NASA launched an unmanned space mission for data-gathering to fly past Venus. It barely made its way out of Cape Canaveral, before veering off course. With a dangerous crash-landing on their hands, NASA activated the self-destruct button. The subsequent investigation revealed that a very simple software error, a missing hyphen in a line of code, led to an estimated loss of 18 Million USD (as of 1962), let alone the loss of face.

EDS Child Support System, 2004, Government of the United Kingdom – In 2004, the UK government introduced a new system to manage the operations of the Child Support Agency (CSA). A Texas-based company called Electronic Data Systems (EDS) was awarded the contract and called it CS2. A plethora of problems plagued it from the initial go-live with over 1,000 reported problems, of which 400 had no known workaround, resulting in around 3,000 IT incidents a week. An internal memo leaked later described the system as badly designed-tested-implemented. Not only did it lead to the UK government spending 768 million GBP instead of the estimated 450 million GBP, but the contractor also announced a fiscal loss of USD 153 million in the subsequent year. 

Bitcoin Hack, Mt. Gox, 2011 – Bacon in the 2010’s Mt Gox was the largest bitcoin exchange in the world until a software glitch led to the creation of unredeemable transactions costing them USD 1.5 Million in lost bitcoins. This severe loss, in an already fluctuating market, led to the company declaring bankruptcy a few years later. 

Considering the above three cases, a clear inference is that the software failed in its ability to satisfy stated or implied needs (as defined by ISTQB) or meet customer or user needs or expectations (as defined by IEEE). Thus comprehensively failing the spirit of both the accepted definitions of Software Quality.

Even though companies spend amounts of money, human resources, and time on QA processes still not being able to achieve the goals is clearly due to a lack of Software Quality.

Even the 12th edition of the World Quality Report 2020-21 lists “QA is a key business priority for organizations to achieve digital transformation”, the Consortium for Information & Software Quality (CISQ) records in its 2020 report that poor quality software costs US companies over USD 2 trillion annually.

Diligent QA Leaders would always seek to ensure optimal software quality for their organizations and set quality goals in the software testing process to avoid such fatal consequences, and this article aims to aid them in their endeavor of Setting SMART goals for Software Quality.

Overview of Software Testing Objectives

Source

The major objectives of any software testing process are outlined in the diagram above. Any attempt to set quality goals for software development will need to take these objectives into account. 

1. Prevent Defects: The most important objective of any Software Quality process is to prevent bugs in the software development process. Early defect detection reduces the costs and labor associated with fixing these defects. By conducting a root cause analysis of previous defects, we can also future-proof our solution against any future flaws.
2. Evaluate Work Product: Any goal of a software quality assurance process demands the creation of artifacts that help set in the context of the product being developed. The principles of agile development demand a shift left in terms of QA, i.e. identifying any ambiguity or contradictory requirements in the requirement document, design, and user stories before development begins to save significant time and cost in the future.
3. Verify Requirements: Meeting the needs of the client is the primary aim of the software development process. Any and all software testing goals gravitate around ensuring that the expected standards are met, and QA techniques are developed to ensure full compliance with both industry standards and user expectations.
4. Validate test objects: All test objects need to be validated in order to ensure that they meet the expectations set by the business requirements of the project. Both manual, as well as automation test procedures, can be implemented to ensure the validity of the test objects.
5. Build Confidence: The bedrock of client confidence in a project is the software QA process. Having a low number of or ideally no defects during user acceptance testing and presenting the clients with comprehensive test reports is a surefire way to build confidence in your product.
6. Reduce Risk: A major goal of any software quality process is to try out as many possible scenarios while testing the product, in the shortest possible time, to identify and root out any risks during deployment.
   • Pro Tip: Use BrowserStack’s cloud Selenium grid of 3000+ real browsers and devices to run all requisite tests in real user conditions and reduce risk. Manual testing is also easily accomplished on the BrowserStack cloud. Sign Up for free, choose the requisite device-browser combinations, and start testing.
7. Share Information to stakeholders: An essential goal for any software quality process is to provide the relevant stakeholders with objective and accurate information to make an educated judgment of the quality of the tested product. Test Coverage reports and other software testing metrics can cover these bases.
   • Pro Tip: Leverage real device testing with BrowserStack. Access 3000+ browsers and devices to test websites and apps. Take advantage of BrowserStack’s comprehensive debugging options to report, record, and resolve bugs.
8. Find Failure and Defects: The primary goal of any software QA process is to test the software as thoroughly as possible to uncover any bugs and ensure superlative software quality.

These software quality objectives provide a useful yardstick for the QA team to evaluate and modify any software testing goals based on their progress. These also enable us to understand the activities which lead to the fulfillment of these objectives to meet our software quality goals.

How to Set Goals for Software Quality

1. Understand what’s most important, get buy-in from your stakeholders

Before we set any goals for software quality, it is important to understand what are the customer’s primary objectives from the software test process. Depending on what the user wants, it is important to ask a variety of questions –

• How long will the testing process take?
• How much money does the test process require?
• How much of the software can be tested through automation?
• Has the test effort been adequate? Could more tests have been executed in the same time frame?

The primary goal for software quality management in software engineering should be to measure software performance across the entire user journey. That means considering the entire user journey from login to checkout. The users should be guaranteed the desired experience at every stage of the user journey.

This stage of the process helps to check the boxes for Building Confidence and Sharing Information with stakeholders from the objectives mentioned above.

2. Define a SMART Goal 

Once the QA Leader has a roadmap for setting the software quality goals, they should set SMART goals to obtain clarity and motivation to complete actionable tasks. By having defined objectives and fixed deadlines, they are more likely to achieve success in what matters rather than just pluck low-hanging fruit.

Source

The SMART acronym stands for –

• Specific: Having a very specific goal. It is best to stick to one very objective aim for every action item
• Measurable: It is important to make sure that the goal set is measurable in some objective terms to evaluate whether it has been achieved. This can be set in terms of code coverage or test coverage. There are also a plethora of software testing metrics that help in quantifying the software testing process. 
• Attainable: The goals created should be realistic. There shouldn’t be any limitations on how big or small it can be, but it should be completed within the time frame determined.
• Relevant: The goal should be important to both the QA team as well as the rest of the stakeholders. This is why the previous step is of utmost importance.
• Time-Bound: It is best to assign an end date to the goal so that it doesn’t get overtaken by the rush of day-to-day activities. Following an agile testing process helps define the workload into manageable sprints which help in fulfilling these objectives. 

Try Testing on Real Device Cloud for Free

Some examples of such quality objectives are:

• “reduce the number of defects by 20% during the next quarter” – assuming that the number of bugs reported during a previous quarter is known;
• “keep re-work time below 25% of the total time spent on development in every sprint“;
• “increase internal testing effectiveness” – targeting a defect detection rate (before vs. after delivery) higher than, let’s say, 80% per release or major milestone.

An example of a bad quality objective could be something vague like “Reduce the number of bugs in every sprint”, as it is not really achievable or time-bound!

Defining SMART goals helps in Evaluating Work Products, Building Confidence, and Sharing Information with stakeholders from the objectives mentioned above.

3. Elucidate the activities required to reach the goal

The activities required to achieve the set goals for software quality can be defined in 3 ways:

Preventive Actions

These include:

• Plans for training your staff on required automation suites or other processes that will improve the overall QA practice.
• R&D activities to ensure the election of the best automation tool, and the adoption of best practices for your QA process.
• Setting up a Continuous Testing process if possible. This ensures that all code is run through a series of tests from the smallest unit to entire components before they are merged to the main repository.

Appraisal Activities 

An appraisal is an objective evaluation of an artifact against a defined set of standards. It consists of –

Reviews 

These are periodic checks of the test process, the progress of test execution, and even the documents created during the software development lifecycle to ensure that the activities being performed on a daily basis are in sync with the user goals and customer requirements. 

Tests 

This is the backbone of any software quality management process. Tests must be performed often and as early as possible. Test automation and parallel testing are two ways to speed up the test process and ensure faster test coverage in minimal time. The entire QA process hinges on the use of a real device cloud. Without real device testing, it is not possible to identify every possible bug a user may encounter. Naturally, undetected bugs cannot be tracked, monitored, or resolved. Moreover, without procuring accurate information on bugs, QA metrics cannot be used to set baselines and measure success. This is true for manual testing and automation testing.

Audits

Audits are usually done by an independent auditor outside of the QA team. They evaluate the work against existing industry standards like ISO standards and so on. Audits need to be planned carefully, defined using a standard model like CMMI, and formalized into a report to share with the stakeholders. Any areas of discrepancy must be converted into a set of SMART goals to ensure full compliance with industry standards.

NOTE: BrowserStack has achieved SOC2 Type 2 compliance, uses HTTPS by default, stores credentials securely, and does much more to ensure the security of our users. Please read our security documentation for detailed descriptions of our policies and controls.

Fix/Repair Activities

These include the efforts to fix the bugs found during testing. A plethora of associated tasks like changing the team structure, testing tool, or test methodology can also be part of this activity.

This step helps to Prevent Defects, Validate test objects, Reduce Risk, Share Information with stakeholders, and Find Failures and Defects from the objectives defined initially.

4. Determine the resources available to see if they are sufficient to match your goals 

There are 3 “actors” in the team who coexist and influence the team dynamic. These are the people (including their skills, availability, and knowledge), the technology (that supports the QA process), and the work processes (waterfall, agile, DevOps, and so on). There must be a balance between these three to ensure the optimal creation and fulfillment of software quality goals. However, there is a “cost” to this balance, and that can be summed up using the following formula –

CoQ = CoP + CoN + CoA (source from American Society for Quality)

• CoP is the cost of prevention resources and activities;
• CoN is the cost of non-conformities: efforts needed to fix defects, re-testing and reworking / refactoring your product; 
• CoA is the cost of appraisals: review time, testing of new or changed functionalities, independent verification, validation, or audit sessions.

A successful quality assurance plan will aim to keep the CoQ within an acceptable range in the overall project’s budget.

5. Time your activities to suit your goals and objectives

Some of the activities described above are once-off activities like Audits, or defining SMART goals. However, others are more continuous. Any processes related to review, verification, and validation will be repetitive across sprints for an agile process for instance.

6. Monitor Quality Objectives continuously

Once the goals have been set, it is important to monitor them continuously in order to ensure that they are relevant to the project and that they are optimal in satisfying the overarching need to adhere to agreed quality standards and user expectations.

Conclusion

In the course of this article, we have defined what software quality means, why is it important for software engineering teams, and how to set about defining goals to meet it. Creating SMART goals for software quality helps keep the process within an agreed time frame and budget, and keeps the team focused on what is most critical for the success of a software project.

When done correctly, the returns from setting Software Quality Goals in terms of team and customer satisfaction, defect reduction, and product quality are undeniable. Add to that the advantage of being seen as acting towards a goal, rather than reacting to what is happening in the SDLC process, which builds customer confidence in the product and leads to a real quality product and not one that barely does what it is set out to do.