I have been doing test automation for quite long time. And I know what pain sometimes deliver new versions of anything. They updated Xcode, released a new Selenium, came up with a new browser (special thanks to Microsoft for Edge and its driver), for some reason, here’s another programming language for you … All this makes the automation tool completely joyful of realizing its own significance. After all, only it is now able to run tests on all of this.

Less experienced guys who only comprehend all the basics and mysteries of working with tests, for some reason, do not experience any joy at all. In our automation course, students are often asked how to work with new versions of various components of the stack. I decided to talk about this today.

The essence of the problem

The main problem is that it is impossible to predict the behavior of the system after changing the version of any component of the stack. The funny thing is, no one will clearly tell you that such a version of one software will not work with such a version of another.

Good example. If you write mobile autotests in Java using Appium, then you know that there are two components: the java_client.jar library generates HTTP requests, and the Appium Server accepts these requests. If you use java_client version 4, and Appium Server is higher than version ten, then the search for elements by ID will fall off. There will be no error, Appium will not send any warings or other signals that something is going wrong. Just the elements will stop to be found. And until you decide to upgrade the java_client version, the tests will not start back. But it was fun to understand all of this for me before! :

A common situation: the new version works with bugs or simply does not cooperate with current versions of other programs. It’s bad if in this case we start randomly updating the remaining components of the stack. This will certainly deliver even more updates and instability.

In inexperienced hands, all of this can lead to anything until the complete breakdown of autotests.

Let’s think about how to work with updates correctly.

Introduction

Let’s say we have a stable working technology stack. In our case, these are Java, JUnit, Java client (Appium client), Appium Server, Selenium library, emulators and simulators (XCode) + the application itself.

A number of tests have already been written on this stack that check the regression of the application before its release. It is important to understand that this is their main task. The release of a broken application can affect its reputation and even lead to financial losses.

New Version Released

Next, one of the components of the stack is updated. This will happen regularly, as our stack includes many programs and libraries.

Do not immediately connect the new version to aproject. This can break (and most likely break) our tests. And this is already regression testing and probably the time of the next release. You have to repair everything in emergency mode, and nobody needs it.

Instead, you first need to decide on the feasibility of an upgrade. To get started, look at the change list. Then read reviews on the Internet: if something is wrong with the release, posts, bug reports or questions in the style of “I had a break down, but did someone else too?” appear very quickly.

Shortly, we answer the question – do we need this update? And if so, how urgent?

Update

When we understand that an upgrade is needed, we create an update ticket, add it to our planning. This is quite the same task as others: writing a new test, making changes to the infrastructure, and so on. At the moment, the update task may have less priority than others in the backlog – this is also normal.

When we get to the ticket, somewhere “on the sidelines” we are collecting a test project, changes in which will not affect the real stack. There we install a new version and try to run tests with it. If necessary, make changes to the code or update dependent components and run the tests again. And so until it works. Moreover, it will not work stably!

Important: in the end, you must definitely run the tests again after adding all new versions and making changes to the code.

All this time, the main stand has been working, fulfilling its task (regression testing and help in releasing the application) on the old stack. This is normal. The process can take more than one week or even a month.

When we are confident in the new stack, it can be transferred to the main stand. It’s better to do this as far as possible from the release in time so that there is still time to roll back.

Summary

You need to think carefully about updates. There are few steps:

  • Examine the list of updates
  • Plan
  • Run separately on the test assembly
  • Only after stabilization of the assembly – transfer to the main stand

The answer to a common question. Is it possible to know in advance which versions will work together and which do not? Unfortunatelly, no. Only empirically build a stack and try to start on it. Even if the developers wrote that it should work, this is no reason not to test in advance. 🙂

I hope this article was useful for someone and will save a couple of hours (or even days) on fixing the falling tests.