At the moment, this is almost the most expensive position in the market. The hustle and bustle around DevOps engineers exceeds all imaginable limits, what to say about Senior DevOps engineers.

I work as
the head of the integration and automation department, according to the English
decryption – DevOps Manager. Whether the English decoding exactly reflects our
daily activities is unlikely, but the other version in this case is more
accurate. By the essence of my activity, it is natural that I need to interview
future members of my team and, over the past year, about 50 people passed
through me.

We are
still looking for colleagues, because a very large layer of various kinds of
engineers is hiding behind the DevOps label.

Everything
written below is my personal opinion, you are not obliged to agree with it, but
I admit that it will bring a touch to your attitude to the topic. Despite the
risk of falling out of favor, I publish my opinion, because I believe that it
has a place to be.

Companies
differently understand who DevOps engineers are and for the sake of quickly
hiring a resource they hang this label for everyone. The situation is rather
strange, because companies are willing to pay unrealistic rewards to these
people, receiving, in most cases, an admin for them.

So who are DevOps
engineers?

Let’s start
with the history – Development Operations appeared as another step towards
optimizing the interaction in small teams to increase the production speed of
the product, as an expected consequence. The idea was to strengthen the
development team with knowledge of procedures and approaches in managing the
product environment. In other words, the developer must understand and know how
his product works under certain conditions, he must understand how to deploy this
product, which environmental characteristics are needed to increase
productivity. So, for some time, developers appeared with the DevOps approach.
DevOps developers wrote build and packaging scripts to simplify their
activities and the performance of a productive environment. However, the complexity
of the decision architecture and the mutual influence of infrastructure
components over time began to worsen the performance of environments, with each
iteration, an ever deeper understanding of those or other components was
required, reducing the productivity of the developer himself due to the
additional cost of understanding the components and tuning systems for a
specific task . The developer’s own cost was growing, the cost of the product
along with it, the requirements for new developers in the team jumped sharply,
because they also had to cover the responsibilities of the “star” of
development and, naturally, the “star” became less accessible. It is also worth
noting that, in my experience, few developers are interested in the specifics
of packet processing by the kernel of the operating system, packet routing
rules, and host security aspects. The logical step was to attract an
administrator who is familiar with this particular responsibility and assign
the responsibility of a similar format to him, which, thanks to his experience,
made it possible to achieve the same indicators at a lower cost in comparison
with the cost of the “star” of development. Such administrators were placed in
a team and their main task was to manage test and productive environments,
based on the rules of a particular team, with resources allocated to this
particular team. So, in fact, DevOps appeared in the majority view.

Partially
or completely, over time, these system administrators began to understand the
needs of this particular team in the development field, how to simplify the
life of developers and testers, how to roll out the update and not stay
overnight at the office on Friday, fixing the deployment errors. Time passed,
now system administrators became the “stars”, they understood what the
developers wanted. In order to minimize impact, management utilities began to
catch up, everyone remembered the old and reliable methods of isolating the OS
level, which allowed minimizing security requirements, managing the network
part, and the host configuration as a whole and, as a result, reduce the
requirements for new “stars”.

A
“wonderful” thing has appeared – docker. Why wonderful? Yes, just because
creating isolation in chroot or jail, as well as OpenVZ, required non-trivial
knowledge of the OS, in a counter utility that allows you to simply create an
isolated application environment on a certain host with everything you need
inside and transfer the task to the development again, and let the system
administrator manage only one host, ensuring its security and high availability
– a logical simplification. But progress does not stand on one place and
systems are becoming more and more complicated, more and more components, one
host no longer satisfies the needs of the system and it is necessary to build
clusters, we again return to system administrators who are able to build these
systems.

Cycle by
cycle, various systems appear that simplify development and / or
administration, orchestration systems appear that, just as long as you do not
need to move away from the standard process, are easy to use. Microservice
architecture also appeared to simplify everything described above – fewer
relationships, easier to manage. In my experience, I did not find a fully
microservice architecture, I would say 50 to 50 – 50 percent of microservices,
black boxes, came in, processed, the other 50 – a monolith, services unable to work
separately from other components. All of this again imposed restrictions on the
level of knowledge of both developers and administrators.

Similar
“swings” of the level of expert knowledge of a particular resource
continue to this day. But we were a little distracted, there are many points
that are worth highlighting.

Build engineer / release
engineer

Very highly
specialized engineers who appeared as a means of standardizing software
assembly processes and its releases. In the process of introducing the Agile,
it would seem that they ceased to be in demand, but this is far from the case.
This specialization appeared as a means of standardization of the assembly and
delivery of software on an industrial scale, i.e. using standard techniques for
all products of the company. With the advent of DevOps, developers have
partially lost their functions, since these were the developers who began to
prepare the product for delivery, and given the changing infrastructure and the
approach to the fastest delivery without regard to quality, over time they
turned into a stopper of changes, since adherence to quality standards
inevitably slows down deliveries. So, gradually, part of the Build / Release
functionality of engineers migrated to the shoulders of system administrators.

Ops’s are so different

We move on
and again the presence of a wide range of responsibilities and the lack of
qualified personnel pushes us to tough specialization, like mushrooms after
rain, various Operations appear:

  • TechOps – enike system administrators aka
    HelpDesk Engineer
  • LiveOps – system administrators primarily
    responsible for productive environments
  • CloudOps – system administrators specializing
    in public “clouds” of Azure, AWS, GCP, etc.
  • PlatOps / InfraOps / SysOps – infrastructure
    system administrators.
  • NetOps – Network Admins
  • SecOps – system administrators specializing in
    information security – PCI compliance, CIS compliance, patching, etc.

DevOps –
(in theory) a person who first-hand understands all the processes of the
development cycle – development, testing, understanding the product
architecture, is able to assess security risks, familiar with automation
approaches and tools, at least at a high level, and also understands the pre-
and post- product release support. A person is able to advocate both Operations
and Development, which allows to build a favorable cooperation between the two
pillars. Understanding team planning processes and managing customer
expectations.

To perform
this kind of work and responsibilities, this person must have the means to
control not only the development, testing, but also management of the product
infrastructure, as well as resource planning. DevOps in this sense cannot be
located either in IT, in R&D, or even in PMO, it must have influence in all
these areas – the technical director of the company, Chief Technical Officier.

Is this so
in your company? – I doubt it. In most cases, it is either IT or R&D.

The lack of
funds and the possibility of influencing at least one of these three lines of
business will shift the weight of the problems to the side where these changes
are easier to apply, such as applying technical restrictions on releases in
connection with dirty code according to the data of the static analyzer
systems. That means, when the PMO sets a tight deadline for the release of
functionality, R&D cannot give a high-quality result within these deadlines
and gives it as it can, leaving refactoring for later, DevOps related to IT, by
technical means blocks the release. The lack of authority to change the
situation, in the case of responsible employees, leads to the manifestation of
hyperresponsibility for what they cannot influence, moreover, if these
employees understand and see mistakes, and how to fix them is “Happiness in
ignorance”, and as a result burnout and loss of these employees.