I have been doing test automation for quite long time. And
I know what pain sometimes deliver new versions of anything. They updated
Xcode, released a new Selenium, came up with a new browser (special thanks to
Microsoft for Edge and its driver), for some reason, here’s another programming
language for you … All this makes the automation tool completely joyful of
realizing its own significance. After all, only it is now able to run tests on
all of this.

Less experienced guys who only comprehend all the basics
and mysteries of working with tests, for some reason, do not experience any joy
at all. In our automation course, students are often asked how to work with new
versions of various components of the stack. I decided to talk about this
today.

The essence of the problem

The main problem is that it is impossible to predict the
behavior of the system after changing the version of any component of the
stack. The funny thing is, no one will clearly tell you that such a version of
one software will not work with such a version of another.

Good example. If you write mobile autotests in Java using
Appium, then you know that there are two components: the java_client.jar
library generates HTTP requests, and the Appium Server accepts these requests.
If you use java_client version 4, and Appium Server is higher than version ten,
then the search for elements by ID will fall off. There will be no error,
Appium will not send any warings or other signals that something is going
wrong. Just the elements will stop to be found. And until you decide to upgrade
the java_client version, the tests will not start back. But it was fun to
understand all of this for me before! :

A common situation: the new version works with bugs or
simply does not cooperate with current versions of other programs. It’s bad if
in this case we start randomly updating the remaining components of the stack.
This will certainly deliver even more updates and instability.

In inexperienced hands, all of this can lead to anything
until the complete breakdown of autotests.

Let’s think about how to work with updates correctly.

Introduction

Let’s say we have a stable working technology stack. In
our case, these are Java, JUnit, Java client (Appium client), Appium Server,
Selenium library, emulators and simulators (XCode) + the application itself.

A number of tests have already been written on this stack
that check the regression of the application before its release. It is
important to understand that this is their main task. The release of a broken
application can affect its reputation and even lead to financial losses.

New Version Released

Next, one of the components of the stack is updated. This
will happen regularly, as our stack includes many programs and libraries.

Do not immediately connect the new version to aproject.
This can break (and most likely break) our tests. And this is already
regression testing and probably the time of the next release. You have to
repair everything in emergency mode, and nobody needs it.

Instead, you first need to decide on the feasibility of an
upgrade. To get started, look at the change list. Then read reviews on the
Internet: if something is wrong with the release, posts, bug reports or
questions in the style of “I had a
break down, but did someone else too?” appear very quickly.

Shortly, we answer the question – do we need this update?
And if so, how urgent?

Update

When we understand that an upgrade is needed, we create an
update ticket, add it to our planning. This is quite the same task as others:
writing a new test, making changes to the infrastructure, and so on. At the
moment, the update task may have less priority than others in the backlog –
this is also normal.

When we get to the ticket, somewhere “on the sidelines” we
are collecting a test project, changes in which will not affect the real stack.
There we install a new version and try to run tests with it. If necessary, make
changes to the code or update dependent components and run the tests again. And
so until it works. Moreover, it will not work stably!

Important: in the end, you must definitely run the tests
again after adding all new versions and making changes to the code.

All this time, the main stand has been working, fulfilling
its task (regression testing and help in releasing the application) on the old
stack. This is normal. The process can take more than one week or even a month.

When we are confident in the new stack, it can be
transferred to the main stand. It’s better to do this as far as possible from
the release in time so that there is still time to roll back.

Summary

You need to think carefully about updates. There are few steps:

  • Examine the list of updates
  • Plan
  • Run separately on the test assembly
  • Only after stabilization of the assembly – transfer to the main stand

The answer to a common question. Is it possible to know in
advance which versions will work together and which do not? Unfortunatelly, no.
Only empirically build a stack and try to start on it. Even if the developers
wrote that it should work, this is no reason not to test in advance. 🙂

I hope this article was useful for someone and will save a
couple of hours (or even days) on fixing the falling tests.