This article will be interesting for both testers and developers, but it is aimed more at automations who are faced with the problem of configuring GitLab CI / CD for integration testing in conditions of insufficient infrastructure resources and / or lack of a container orchestration platform. I’ll tell you how to configure the deployment of test environments using Docker compose on a single GitLab shell runner so that during deploying multiple environments, the services being launched do not interfere with each other.

Preconditions

In my practice, it often happened to “heal” integration testing on projects. And often the first and most significant problem is the CI pipeline, in which integration testing of the developed service (s) is carried out in a dev / stage environment. This caused quite a few problems:

  • Due to defects in a particular service, the test circuit may be corrupted by broken data during integration testing. There were cases in which sending a request with a broken JSON format damaged a service, which made the stand completely non-working.
  • Slowing down the operation of the test circuit with the growth of test data. I think it makes no sense to describe an example with cleaning / rolling back a database. In my practice, I have not met a project where this procedure would go smoothly.
  • Risk of disruption of the test circuit while testing general system settings. For example, user / group / password / application policy.
  • Test data from autotests is disturbing manual testers.

Someone will say that good autotests should clean the data after themselves. I have arguments against:

  • Dynamic stands are very convenient to use.
  • Not every object can be deleted from the system through the API. For example, a call to delete an object is not implemented, because it contradicts business logic.
  • When creating an object through the API, can be created a huge amount of metadata, which is problematic to delete.
  • If the tests are interdependent, then the data cleaning process after the tests are completed turns into a headache.
  • Additional (and, in my opinion, not always needed) calls to the API.

And the main argument: when the test data begins to clean directly from the database. It turns into a real circus! From the developers you can hear: “I only added / deleted / renamed the table, why did 100500 integration tests fall?”

In my opinion, the most optimal solution is a dynamic environment.

Many people use docker-compose to run a test environment, but few use docker-compose when conducting integration testing in CI / CD. And here I do not take into account kubernetes, swarm and other container orchestration platforms. Not every company has them. It would be nice if docker-compose.yml was universal.

  • Even if we have our own QA runner, how can we make sure that services launched via docker-compose do not interfere with each other?
  • How to collect logs of tested services?
  • How to clean the runner?

I have my own GitLab runner for my projects and I came across these issues while developing a Java client for TestRail. Or rather, when running integration tests. So, we will solve these issues with examples from this project.

Gitlab shell runner

For the runner, I recommend a Linux virtual machine with 4 vCPU, 4 GB RAM, 50 GB HDD.

In the Internet is a lot of information about configuring gitlab-runner, so shortly:

  • We go to the machine on SSH
  • If you have less than 8 GB of RAM, then I recommend making a swap of 10 GB to don’t make the OOM killer come and kill us due to lack of RAM. This can happen when more than 5 tasks are started simultaneously. Tasks will be slower, but stable.
  • Install gitlab-runner, docker, docker-compose, make.
  • Add user gitlab-runner to the docker group
sudo groupadd docker
sudo usermod -aG docker gitlab-runner
  • Register gitlab-runner.
  • Open for editing /etc/gitlab-runner/config.toml and add
concurrent=20
[[runners]]
  request_concurrency = 10

This will allow you to run parallel tasks on the same runner.

If your machine is more powerful, for example, 8 vCPU, 16 GB RAM, then these numbers can be at least 2 times larger. But it all depends on what exactly will be launched on this runner and in what quantity.

It’s enough.

Preparing of docker-compose.yml

The main task is docker-compose.yml, which will be used both locally and in the CI pipeline.

The variable COMPOSE_PROJECT_NAME will be used to start several instances of the environment.

An example of my docker-compose.yml:

version: "3"

volumes:
  static-content:

services:
  db:
    image: mysql:5.7.22
    environment:
      MYSQL_HOST: db
      MYSQL_DATABASE: mydb
      MYSQL_ROOT_PASSWORD: 1234
      SKIP_GRANT_TABLES: 1
      SKIP_NETWORKING: 1
      SERVICE_TAGS: dev
      SERVICE_NAME: mysql

  migration:
    image: registry.gitlab.com/touchbit/image/testrail/migration:latest
    links:
    - db
    depends_on:
    - db

  fpm:
    image: registry.gitlab.com/touchbit/image/testrail/fpm:latest
    container_name: "testrail-fpm-${CI_JOB_ID:-local}"
    volumes:
    - static-content:/var/www/testrail
    links:
    - db

  web:
    image: registry.gitlab.com/touchbit/image/testrail/web:latest
  
    ports:
      - ${TR_HTTP_PORT:-80}:80
      - ${TR_HTTPS_PORT:-443}:443
    volumes:
      - static-content:/var/www/testrail
    links:
      - db
      - fpm

 

Makefile Preparation

I use Makefile, as it is very convenient both for local management of the environment, and in CI.

ifeq ($(CI_JOB_ID),)
  CI_JOB_ID := local
endif

export COMPOSE_PROJECT_NAME = $(CI_JOB_ID)-testrail
docker-compose -f .indirect/docker-compose.yml down

docker-up: docker-down
    
    docker-compose -f .indirect/docker-compose.yml pull
   
    docker-compose -f .indirect/docker-compose.yml up --force-recreate --renew-anon-volumes -d
    

docker-logs:
    mkdir -p ./logs
    docker logs $${COMPOSE_PROJECT_NAME}_web_1       >& logs/testrail-web.log       || true
    docker logs $${COMPOSE_PROJECT_NAME}_fpm_1       >& logs/testrail-fpm.log       || true
    docker logs $${COMPOSE_PROJECT_NAME}_migration_1 >& logs/testrail-migration.log || true
    docker logs $${COMPOSE_PROJECT_NAME}_db_1        >& logs/testrail-mysql.log     || true


docker-clean:
    @echo 
    docker kill $$(docker ps --filter=name=testrail -q) || true
    @echo 
    docker rm -f $$(docker ps -a -f --filter=name=testrail status=exited -q) || true
    @echo 
    docker rmi -f $$(docker images -f "dangling=true" -q) || true
    @echo 
    docker rmi -f $$(docker images --filter=reference='registry.gitlab.com/touchbit/image/testrail/*' -q) || true
    @echo 
    docker volume rm -f $$(docker volume ls -q) || true
    @echo 
    docker network rm $(docker network ls --filter=name=testrail -q) || true
    docker ps

Preparing of .gitlab-ci.yml

Run of integration tests

Integration:
  stage: test
  tags:
    - my-shell-runner
  before_script:
    - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY}
    - export TR_HTTP_PORT=$(shuf -i10000-60000 -n1)
    - export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1)
  script:
    - make docker-up
    - java -jar itest.jar --http-port ${TR_HTTP_PORT} --https-port ${TR_HTTPS_PORT}
    - docker run --network=testrail-network-${CI_JOB_ID:-local} --rm itest
  after_script:
    - make docker-logs
    - make docker-down
  artifacts:
    when: always
    paths:
      - logs
    expire_in: 30 days

As a result of launching such a task in the artifacts, the logs directory will contain the logs of services and tests. Which is very convenient in case of errors. Each of my tests in parallel writes its own log, but I will talk about this separately.

Runner Cleaning

The task will be launched only on schedule.

stages:
- clean
- build
- test

Clean runner:
  stage: clean
  only:
    - schedules
  tags:
    - my-shell-runner
  script:
    - make docker-clean

Next, go to our GitLab project -> CI / CD -> Schedules -> New Schedule and add a new schedule.

Result

Run 4 tasks in GitLab CI.

In the logs of the last task with integration tests, we see containers from different tasks.

CONTAINER ID  NAMES
c6b76f9135ed  204645172-testrail-web_1
01d303262d8e  204645172-testrail-fpm_1
2cdab1edbf6a  204645172-testrail-migration_1
826aaf7c0a29  204645172-testrail-mysql_1
6dbb3fae0322  204645084-testrail-web_1
3540f8d448ce  204645084-testrail-fpm_1
70fea72aa10d  204645084-testrail-mysql_1
d8aa24b2892d  204644881-testrail-web_1
6d4ccd910fad  204644881-testrail-fpm_1
685d8023a3ec  204644881-testrail-mysql_1
1cdfc692003a  204644793-testrail-web_1
6f26dfb2683e  204644793-testrail-fpm_1
029e16b26201  204644793-testrail-mysql_1
c10443222ac6  204567103-testrail-web_1
04339229397e  204567103-testrail-fpm_1
6ae0accab28d  204567103-testrail-mysql_1
b66b60d79e43  204553690-testrail-web_1
033b1f46afa9  204553690-testrail-fpm_1
a8879c5ef941  204553690-testrail-mysql_1
069954ba6010  204553539-testrail-web_1
ed6b17d911a5  204553539-testrail-fpm_1
1a1eed057ea0  204553539-testrail-mysql_1

It seems everything is beautiful, but there is a nuance. Pipeline can be forced to cancel during the integration tests, in which case running containers will not be stopped. From time to time you need to clean the runner. Unfortunately, the revision task in GitLab CE is still in Open status.

But we have added the scheduled task launch, and no one forbids us to start it manually.

Go to our project -> CI / CD -> Schedules and run the Clean Runner task:

Total:

  • We have one shell runner.
  • There are no conflicts between tasks and the environment.
  • We have a parallel launch of tasks with integration tests.
  • You can run integration tests both locally and in the container.
  • Service and test logs are collected and attached to the pipeline task.
  • It is possible to clean the runner from old docker-images.

Setup time is ~ 2 hours.

We will be glad to get you feedback, for this don’t hesitate to contact us.