Do we need an auto-testing? When is it needed? What value does it bring?

The article discusses when and why testing is needed and in which cases its automation would be useful.

Test automation is needed if it brings value

 When does testing itself bring value? Two cases have been identified.

  • If the process catches errors in the software before production
  • If testing the new version revealed errors that were subsequently corrected, then this testing was made well.

But what if the situation is reversed? If the testing did not find an error, but they showed up in production? If errors were detected on the test bench (including the test bench configured most closely to the production).

In this case the testing was carried out poorly.

How to measure the quality of testing? For this case, is suitable the metric = the number of errors on the test environment / (the number of errors on the test + main environment). In this case, the number of errors is taken weighted by the level of criticality.

But what if the testing did not find errors and they were not found on the main server?

In this case, testingdid not bring value and these works were done for nothing (with the exception of the following case, which we will discuss later). From general point of view, this is a loss.

When is this possible? When the module under test did not change. What can change a module?

  • Modification of the module code.
  • Changing the version of the libraries used (including OS, database, etc.). The change may be due to the requirements of regulators, that’s why the library needs to be updated in a fixed time frame.
  • Changing settings or data that seriously affects the behavior of a universal functional, the comprehensive testing of which is unnecessarily difficult to conduct, and as a result, testing was abandoned. Examples:
  • Promotional settings in solutions like Siebel CRM or Oracle ATG can degrade the performance of the promotional calculation functionality, and the possibility of comprehensive verification is not possible in a reasonable time due to the excessive flexibility and universality of these solutions
  • The html description of the product card may contain a broken document structure or errors in the js code description written inside, which breaks the product card page
  • The use of the functional is not intended for this purpose. For example, a change in the type of load that is not inherent in the requirements and, as a result, is not taken into account during the testing. For example, before Black Friday, it is worthwhile to conduct a separate load testing for the landing page, where will the traffic go, if there were no such load requirements for this type of page before.
  • Changing the behavior of the API of other modules that the module under development uses. Especially if the API functionality is not covered by regression testing.

Since the options for changes may be different, and conducting a full regression test costs money, it is not worth to make all the tests. One control option is to tag tests before testing. The test manager determines which set of tests should be sent for execution for a given portion of changes.

When do I need to write autotests in this case?

First of all, automated testing doesn’t negate setting of testing, design and writing test setting cases! And doesn’t replace them. If this is not the case, then automation should not be dealt with. At the same time, autotests should be understood not only as the script itself, but also as preparation for their run and use of the results.

If you write self-tests after creating the code, this will lead to an increase in time2market (which will automatically lead to an increase in connected capital). Therefore, if it is decided to cover the code with autotests, then you should write these autotests in parallel to the main code, in the development paradigm of “TestFirst” or “TDD”.

The main value created by the automation of testing is the reduction of time2market due to the faster release of the new version.

Tests are needed to guarantee the performance of critical processes

Despite the fact that the car never caught fire, the presence of a fire extinguisher in it is not useless.

An error on a site with high traffic that doesn’t allow adding products to the cart can lead to more significant losses than the cost of developing and conducting testing of this function for the year.

Therefore, it is necessary to highlight the critical processes that will go into regular checks (which are worth doing if some kind of change occurs). Compare:

  • loss from downtime of the function from the time of its detection to correction,
  • expenses for manual regular testing or its automation and its subsequent implementation during the payback period.

But what if regular testing doesn’t find errors for a long time and they do not occur on production? Then testing doesn’t bring value, and therefore it is not necessary. When it’s possible?

  • The module under development is not very large.
  • Stable, highly competent team.
  • Integrations are closed by tests or on the other side there are no changes.

Is it possible not to do testing at all?

This is possible through the launch of several installations of the solution and testing of new beta versions, if this is technically possible and if such volunteers are found. After publishing the new version, is monitored a telemetry and performed a rollback, in case of degradation of indicators.

Another useful case of regression auto-testing is API testing (API contract testing), if this API is required to support a critical process. This is especially important if the developers of another module change something and do not do quality testing of changes on their side.

When test automation is not needed

If you got a large amount of inherited not very high-quality code. Covering autotests with such chaos is increasing chaos.

In this case, it is worth highlighting the logical module from this solution. After selecting the interaction layer of this module with the rest of the code, you need to cover the interaction with autotests. This will provide guarantees of the integrity of the module’s behavior and integrity after its transcoding.

Autotests do not replace manual testing. Manual testing is most often carried out faster and cheaper than writing and subsequently accompanying autotests. In particular, if testing is carried out after the development of the code, then from this testing only the piece that will go into the regular testing of critical functionality should be automated.

And finally – a checklist of the company’s readiness for autotests.

Let’s make a reservation right away, this checklist is not for everyone, it is written for testers of companies for which software development is the main source of income.

Check list of the company’s readiness for autotests

  1. The company has well-made the main business delivery process and there is an understanding of where you are located.
  2. The company has made up the process of delivering value to customers.
  3. Task management has been set up, which means that all involved people take tasks to the desired status. And the tasks are typified.
  4. The company has formulated testing goals.
  5. The headings of the tasks in the tracker are “clear”, in other words, by the title you can understand what kind of task it is.
  6. The task registry is manageable, at any time it reflects the current status and policy of the project / product.
  7. There is a registry of requirements and it is manageable.
  8. There is traceability of task requirements.
  9. There is traceability of test requirements. It is known what requirements are covered by what tests.
  10. There is traceability of task tests. It is known what has already been tested where and how.
  11. There is an influence matrix of the components on each other.
  12. Tasks are traced to components.
  13. Everything is on version control.
  14. There is a versioned policy of who, how and why. There is an understanding of why git flow is a bad model.
  15. Existing standards: coding and other
  16. There is CI
  17. There is a release policy, where are prescribed methods of versioning
  18. There is a repository for artifacts from where you can uniquely take out a product ready for installation.
  19. There is a policy for marking artifacts according to different criteria. Static code analysis is not forgotten.
  20. The environment is also on version control.
  21. The environment is fully automated and checked for correctness. Ports, Java version and so on.
  22. Product can be easily unfolded.
  23. The product is configured fully automatically for the required task.
  24. You have a way to repeatedly and automatically generate all the necessary test data. Generation scripts are also on version control and associated with product artifacts.
  25. All of the above works for any version of the product.
  26. You have a delivery pipeline registered inside the release policy.