Group 1 – Requirements for the developed software

 

This group of metrics will allow you to evaluate how we have followed the requirements (user story) for software, identify vulnerabilities and the most complex, potentially problematic features of the software, understand where special control is required.

 

1. Test coverage requirements

“Total number of tests” / “Total number of requirements”

 

Purpose of the metric: to identify weaknesses in the test coverage, highlight the risks.

 

This metric will work only if the requirements are well decomposed into equivalent ones. Otherwise, it will turn into an indicator of the presence or absence of tests for each of the requirements.

For requirements in which the coefficient is equal to or close to 0, you need to think of making additional tests.

 

2. The degree of interconnectedness of requirements

AVERAGE (“The number of requirements related to requirement No. 1” / “The total number of requirements -1”, …, “The number of requirements associated with requirement No. n” / “The total number of requirements -1”)

 

The metric is calculated as the number of links of each requirement with the rest of the requirements. In this case is used the average value for all requirements.

 

Purpose of the metric: to provide a basis for assessing the timing of testing and taking into account possible risks. Knowing the degree of influence of requirements on each other, you can, for example, plan additional time and cases for end-to-end testing, make regression checks, think of integration, etc.

 

From our own experience, we can note that an acceptable degree of connectivity does not exceed 0.2-0.3. Otherwise, correction of one of the requirements will lead to a correction chain, and therefore possible errors in a significant part of the product.

3. The coefficient of stability requirements

“Number of changes in existing requirements” / “Total number of requirements implemented during iteration, including new ones”

 

Purpose of the metric: to show how many already implemented requirements have to be redone from release to release by developing new features. Also, the metric gives an idea of ​​how easily the functionality of the system scales and new features can be added.

 

For this metric, the coefficient must be at least less than 0.5. In this case, we make 2 times more new features than redo existing ones. Otherwise, the team focuses more on remaking previously released features, rather than creating new values ​​for the business.

 

Group 2 – Quality of the developed product

 

This group of metrics allows you to evaluate and compare from release to release both the quality of the software and the quality of the development itself.

 

1. Density of defects

“Number of defects in a separate module” / “Total number of defects in a software”

It is calculated as a part of defects from the total number that exist in a separate module of iteration or release.

Purpose of the metric: highlight which part of the software is the most problematic. This information will help in the assessment and planning of work with this module, as well as in risk analysis.

 

This metric will help to concentrate on the problem module \ system \ functionality, where the part of defects is above average.

 

2. The regression coefficient

“The number of defects in the old functional” / “The total number of defects, including the new functional”

 

The purpose of the metric is to show what the team’s efforts are spent on – whether it is more engaged in the development and debugging of new features or spends most of the time fixing in existing parts of the software.

 

For example, if the regression coefficient is more than 0.5, this means that we spend more than half the time restoring previously working software functions. This situation needs to be corrected.

 

3. The coefficient of re-discovered defects

“Number of re-discovered defects” / “Total number of errors, including previously corrected and new”

 

Purpose of the metric: to assess the quality of development and correction of defects, as well as the complexity of the product or a separate module.

 

The closer the coefficient is to 0, the less old errors are repeated.

Moreover, if the value is more than 0.2-0.3, this can indicate either the technical complexity of the module, or problems in the architecture, or even poor-quality error correction.

Group 3 – QA Team capabilities and effectiveness

 

The main task of this group of metrics is to show in numbers what the testing team is able to do. Such indicators should be calculated and compared on a regular basis, as well as analyzing of trends, observing how certain changes affect the work of the team.

 

1. The velocity of the QA team

“The number of story points for N iterations” / “N”

It is calculated as the ratio of realized story points requirements \ user stories for several iterations \ sprints to the number of selected iterations.

Purpose of the metric: to quantify the capabilities, speed of the team work for further planning of the work scope and analysis of development trends. The metric allows you to monitor the speed of QA, to observe which internal or external influences on the team affect this speed.

 

2. The average lifetime of the defect

“Total time to fix found defects” / “Number of defects”

 

The total time during which defects found during the iteration or release to the total of defects.

Purpose of the metric: to show how much, on average, it takes to work with one defect: its registration, correction and reproduction. This indicator will allow you to evaluate the time required for testing and to identify areas of software in which the greatest difficulties arise.

 

A defect’s lifetime is the entire time from its registration to closure minus all possible work pauses. By showing defects with the longest correction time, the metric will help to identify particularly complex and problematic modules or a weakness in the development team.

Group 4 – Quality of the testing team work

 

The goal of this set of metrics is to assess how well testers fulfill their tasks, determine the level of competencies and maturity of the QA team. With this set of indicators, you can compare a team with itself on different time periods or with other, external testing groups.

 

1. The effectiveness of tests and test sets

“Number of detected errors” / “Number of cases in the test set”

 

Purpose of the metric: to show how many errors on average our cases allow to be detected. This metric reflects the quality of the test design and helps to monitor the trend of its change.

 

This indicator of tests allows you to monitor the effectiveness of each of the test sets and its changes over time. This will make possible to supplement them with “fresh” tests on time.

2. The coefficient of errors skipped on the production

“Number of errors detected after publication of the release” / “Total number of errors detected before and after the release”

 

Purpose of the metric: to demonstrate the quality of testing and the effectiveness of error detection – what part of defects was filtered out and which passed to the production.

 

Of course, the permissible percentage of missed errors will depend on many factors, however, if it is> 0.1, then there is clearly a problem, because in this case every tenth defect came to the production and potentially can lead to software crashes among users.

 

3. The real runtime of the QA team

“Time spent on targeted QA activities” / “Total number of working hours of the team”

 

The ratio of the time spent by the team directly on the target QA activity, to the total number of hours.

 

Purpose of the metric: firstly, to increase the accuracy of planning, and secondly, to monitor and manage the effectiveness of the team.

 

Targeted activities may include: analysis, design, evaluations, testing, working meetings, and much more, non-targeted ones – pauses because of blockers, communication problems, inaccessibility of resources, etc.

Of course, this metric will never be equal to 1. Practice shows that for effective teams its value can be 0.5-0.6.

 

4. Accuracy of time estimation by regions / types / kinds of work

Estimated Runtime / Actual Runtime

 

Purpose of the metric: allows you to use the correction factor for the future estimates.

 

The degree of the assessment accuracy can be determined for the entire team or individual testers, full system or individual software modules.

Group 5 – Feedback and User Satisfaction

 

And finally, a group of metrics, which shows how the product was accepted by end users, how it met their expectations. At the same time, as part of evaluating user interactions, it’s important not only to know feedback about the software itself. Another significant task of this group of metrics is to show whether users are satisfied with the process of interaction with the IT team in general and QA in particular.

 

1. Customer satisfaction with IT service

Should be done a regular survey of user satisfaction with the IT service with scoring.

 

Purpose of the metric: to show whether users trust the IT team, if they understand how and why its work is organized, how much this work lives up to expectations.

 

The metric can be a great indicator and show if it is necessary to focus on optimizing the process or to make it more clear and transparent for users.

The satisfaction indicator can be calculated based on the survey results of software delivery. To do this, you need to collect all the grades and calculate the average score.

 

2. Product user satisfaction

A regular survey of users about how satisfied they are with the quality of the product.

Purpose of the metric: to determine how much the developed product meets the expectations of users, whether we are moving in the right direction and correctly determine the importance of features, solutions.

 

To calculate this metric, we also make a survey of users and calculate the average score. By calculating this indicator on a regular basis, you can monitor the trend of user satisfaction.

 

3. Stakeholder involvement

The number of initiatives and proposals for improving the process and product received during the iteration (release) by stakeholders.

 

Purpose of the metric: to determine the degree of participation of external stakeholders (business, infrastructure, users, support, etc.) in the work on the product. Having such a metric on hand, you can understand where to get feedback and don’t face problems and misunderstanding.