Net Quality Score Overview

The Net Quality Score is a proprietary scoring mechanism that reflects the Quality of your product, processes, and system’s health.

The Net Quality Score is a comprehensive metric across different SDLC areas that affect Software Quality. It is calculated based on four factors: Coverage, Quality, Automation, and Velocity. Each factor impacts the score, measured on a scale of 0 to 5.

Factors and Weights

The Net Quality Score (NQS) is a weighted average of scores from individual factors. The weightage for the different factors behind NQS is as follows.

Coverage

The Net Quality Score tracks the automation coverage of your test cases as part of the Coverage factor. This factor has a high impact on the NQS. You can connect and configure test management tools to start measuring this factor.

Why is this critical?

The Coverage factor helps you determine the readiness of your test suite and its automation status. Test automation coverage refers to the extent to which automated tests cover various features and functionalities of a software application. It measures the effectiveness of automated testing in ensuring that critical paths and scenarios are tested thoroughly.

How is this measured?

The coverage factor is measured using the two metrics below.

1. Automation Coverage - The percentage of your test cases that are automated.
2. Automation Coverage of Critical Tests - The percentage of your critical test cases that are automated. The criticality of test cases refers to the priority detected on your test management tool. Critical and High priority tests’ automation status are considered critical tests covered.

How to improve this factor?

Prioritizing the automation work of pending manual test cases can improve the Coverage factor. Use the Quality Engineering Insights tool to localize the manual test cases and prioritize the automation work for critical test cases.

Quality

The Net Quality Score tracks the effectiveness and correctness of testing processes as part of the Quality factor. This factor has a very high impact on the NQS. You can connect and configure your issue tracker tools like JIRA to start measuring this factor.

Why is this critical?

The Quality factor helps you determine the effectiveness of the testing process during the software delivery. It measures the effectiveness of testing the software and ensures that it is delivered to customers with minimal defects in production.

How is this measured?

The Quality factor is measured by the three metrics below.

1. Defect Leakage - The percentage of defects that leaked to production.
2. Leakage of Critical Defects - The percentage of critical defects that leaked to production. The criticality of test cases refers to the defect priority detected on your issue tracking tool.
3. Defect Rejection Rate - The percentage of rejected defects during the testing process.

How to improve this factor?

Continuously improving the test suite and ensuring automation for manual test cases can enhance the Quality factor. Use the Quality Engineering Insights tool to understand what areas are generating defects and what defect types are being leaked to production. Localize why these defects are not caught in non-production environments.

Automation

The Net Quality Score tracks the health and efficiency of your CICD Automation setup as part of the Automation factor. This factor has a high impact on the NQS. To start measuring this factor, you can connect and configure your CI/CD tools like Jenkins or test reporters like Test Observability.

Why is this critical?

The Automation factor helps you determine the health of your testing jobs configured on your CICD setup. It measures the maturity of your automation setup and ensures that software is delivered to customers with no to minimal defects and without any production delays.

How is this measured?

The below metrics measure the Automation factor.

1. Average Build Stability - This metric tracks the test passing percentage across different CI jobs. The lower the number of failing or timed-out tests, the higher the build stability.
2. Average Build Performance - This metric tracks the total execution time of CI jobs. The recommendation is to have a threshold of 3 hours for all of your CI jobs. If the execution time for CI jobs is higher, this metric is affected.
3. Job Frequency - This metric tracks the frequency of your CI jobs. This metric is affected if the jobs configured are stale and run infrequently.
4. Build Flakiness - This metric tracks the stability of test cases configured as part of your CI jobs. If the test cases are flaky, this metric is affected.

How to improve this factor?

Continuously monitoring and improving the automated tests can improve the automation factor. Use the Quality Engineering Insights tool to understand what jobs affect the overall automation-related metrics.

Velocity

The Net Quality Score tracks the effectiveness of your testing and deployment processes while building new features as part of the Velocity factor. This factor has a medium impact on the NQS. You can connect and configure your project management tools like JIRA to start measuring this factor.

Why is this critical?

The Velocity factor helps you determine how the current test coverage and automation setup are helping you deliver software. It measures the maturity of your testing processes and ensures the testing systems are helping reduce time to production.

How is this measured?

The below metrics measure the Velocity factor.

1. Number of stories with either Dev <> QA or Dev <> Review iterations - This metric tracks the number of stories iterating back and forth during development due to a lack of testing jobs or poor development quality.
2. Percentage time in testing and deployment - This metric tracks the total time spent by the story in testing and deployment. The lack of automation or CICD setup affects the metric.

How to improve this factor?

Introducing testing processes at the development stage can improve the velocity factor. Better test case planning and the right test pyramid with a mix of unit testing and end-to-end testing ensure defects are detected early in the cycle. Having automated jobs running post-development improves the overall speed of delivering software. Use the Quality Engineering Insights tool to understand what teams need the correct setup to affect their Cycle time.

Have any Feedback?

Do you have any feedback about the Net Quality Score and how we can improve it? Please reach out to us here.