Do we need an auto-testing? When is it needed? What value does it bring?

The article discusses when and why testing is needed and in which cases
its automation would be useful.

Test automation is needed if it brings value

 When does testing itself bring value? Two cases have been identified.

  • If the process catches errors
    in the software before production
  • If testing the new version
    revealed errors that were subsequently corrected, then this testing was made
    well.

But what if the situation is reversed? If the testing did not find an
error, but they showed up in production? If errors were detected on the test bench
(including the test bench configured most closely to the production).

In this case the testing was carried out poorly.

How to measure the quality of testing? For this case, is suitable the
metric = the number of errors on the test environment / (the number of errors
on the test + main environment). In this case, the number of errors is taken
weighted by the level of criticality.

But what if the testing did not find errors and they were not found on
the main server?

In this case, testingdid not bring value and these works were done for
nothing (with the exception of the following case, which we will discuss
later). From general point of view, this is a loss.

When is this possible? When the module under test did not change. What
can change a module?

  • Modification of the module
    code.
  • Changing the version of the
    libraries used (including OS, database, etc.). The change may be due to the
    requirements of regulators, that’s why the library needs to be updated in a
    fixed time frame.
  • Changing settings or data that
    seriously affects the behavior of a universal functional, the comprehensive
    testing of which is unnecessarily difficult to conduct, and as a result,
    testing was abandoned. Examples:
  • Promotional
    settings in solutions like Siebel CRM or Oracle ATG can degrade the performance
    of the promotional calculation functionality, and the possibility of
    comprehensive verification is not possible in a reasonable time due to the
    excessive flexibility and universality of these solutions
  • The
    html description of the product card may contain a broken document structure or
    errors in the js code description written inside, which breaks the product card
    page
  • The use of the functional is
    not intended for this purpose. For example, a change in the type of load that
    is not inherent in the requirements and, as a result, is not taken into account
    during the testing. For example, before Black Friday, it is worthwhile to
    conduct a separate load testing for the landing page, where will the traffic go,
    if there were no such load requirements for this type of page before.
  • Changing the behavior of the
    API of other modules that the module under development uses. Especially if the
    API functionality is not covered by regression testing.

Since the options for changes may be different, and conducting a full
regression test costs money, it is not worth to make all the tests. One control
option is to tag tests before testing. The test manager determines which set of
tests should be sent for execution for a given portion of changes.

When do I
need to write autotests in this case?

First of all, automated testing doesn’t negate setting of testing,
design and writing test setting cases! And doesn’t replace them. If this is not
the case, then automation should not be dealt with. At the same time, autotests
should be understood not only as the script itself, but also as preparation for
their run and use of the results.

If you write self-tests after creating the code, this will lead to an
increase in time2market (which will automatically lead to an increase in
connected capital). Therefore, if it is decided to cover the code with
autotests, then you should write these autotests in parallel to the main code,
in the development paradigm of “TestFirst” or “TDD”.

The main value created by the automation of testing is the reduction of
time2market due to the faster release of the new version.

Tests
are needed to guarantee the performance of critical processes

Despite the fact that the car never caught fire, the presence of a fire
extinguisher in it is not useless.

An error on a site with high traffic that doesn’t allow adding products
to the cart can lead to more significant losses than the cost of developing and
conducting testing of this function for the year.

Therefore, it is necessary to highlight the critical processes that will
go into regular checks (which are worth doing if some kind of change occurs).
Compare:

  • loss from downtime of
    the function from the time of its detection to correction,
  • expenses for manual
    regular testing or its automation and its subsequent implementation during the
    payback period.

But what if regular testing doesn’t find errors for a long time and they
do not occur on production? Then testing doesn’t bring value, and therefore it
is not necessary. When it’s possible?

  • The module under
    development is not very large.
  • Stable, highly
    competent team.
  • Integrations are closed
    by tests or on the other side there are no changes.

Is
it possible not to do testing at all?

This is possible through the launch of several installations of the
solution and testing of new beta versions, if this is technically possible and
if such volunteers are found. After publishing the new version, is monitored a telemetry
and performed a rollback, in case of degradation of indicators.

Another useful case of regression auto-testing is API testing (API
contract testing), if this API is required to support a critical process. This
is especially important if the developers of another module change something
and do not do quality testing of changes on their side.

When
test automation is not needed

If you got a large amount of inherited not very high-quality code.
Covering autotests with such chaos is increasing chaos.

In this case, it is worth highlighting the logical module from this
solution. After selecting the interaction layer of this module with the rest of
the code, you need to cover the interaction with autotests. This will provide
guarantees of the integrity of the module’s behavior and integrity after its
transcoding.

Autotests do not replace manual testing. Manual testing is most often
carried out faster and cheaper than writing and subsequently accompanying
autotests. In particular, if testing is carried out after the development of
the code, then from this testing only the piece that will go into the regular
testing of critical functionality should be automated.

And finally – a checklist of the company’s readiness for autotests.

Let’s make a reservation right away, this checklist is not for everyone,
it is written for testers of companies for which software development is the
main source of income.

Check
list of the company’s readiness for autotests

  1. The company has well-made the main business delivery process and there
    is an understanding of where you are located.
  2. The company has made up the process of delivering value to customers.
  3. Task management has been set up, which means that all involved people take
    tasks to the desired status. And the tasks are typified.
  4. The company has formulated testing goals.
  5. The headings of the tasks in the tracker are “clear”, in other words, by
    the title you can understand what kind of task it is.
  6. The task registry is manageable, at any time it reflects the current
    status and policy of the project / product.
  7. There is a registry of requirements and it is manageable.
  8. There is traceability of task requirements.
  9. There is traceability of test requirements. It is known what
    requirements are covered by what tests.
  10. There is traceability of task tests. It is known what has already been
    tested where and how.
  11. There is an influence matrix of the components on each other.
  12. Tasks are traced to components.
  13. Everything is on version control.
  14. There is a versioned policy of who, how and why. There is an
    understanding of why git flow is a bad model.
  15. Existing standards: coding and other
  16. There is CI
  17. There is a release policy, where are prescribed methods of versioning
  18. There is a repository for artifacts from where you can uniquely take out
    a product ready for installation.
  19. There is a policy for marking artifacts according to different criteria.
    Static code analysis is not forgotten.
  20. The environment is also on version control.
  21. The environment is fully automated and checked for correctness. Ports,
    Java version and so on.
  22. Product can be easily unfolded.
  23. The product is configured fully automatically for the required task.
  24. You have a way to repeatedly and automatically generate all the
    necessary test data. Generation scripts are also on version control and
    associated with product artifacts.
  25. All of the above works for any version of the product.
  26. You have a delivery pipeline registered inside the release policy.