DevOps Time for Java Developers
The topic for discussion is building Java applications in CI.
Let’s take a look at the Maven repository on Docker Hub. The latest version (3.8.1) appears everywhere, but images based on Alpine are only ibmjava-8, there are no 11 or current releases. The 11-slim image takes 230 MB (compressed size). In our projects, we use BellSoft Liberica (current lts / release) Alpine Musl as a base image for applications. It is known that some teams sometimes have problems with Alpine and Musl when solving specific tasks, but this never concerned us (only the limitation on the size of the space in the provided Container Registry concerned, and this was a plus for choosing Alpine). The Java 11 image takes up only 75 MB (compressed size) – only 30% of the 11-slim size. But it doesn’t have Maven.
All developed applications use Spring and Testcontainers are regularly encountered in them. Often this means that the docker daemon socket has already been forwarded to the container by the system administrators. If the image had the docker command-line utility, it would be possible to immediately build and push the image, and not transfer it through the GitLab artifacts to the next stage, wait for the docker: stable download, and thereby avoid various overheads. Building the image immediately after receiving the .jar would be useful for applications with a short build time, but if your unit tests take a lot of time and the Container Registry sometimes crashes, it’s better not to do this.
Gradually, you come to the conclusion that it would be most convenient to have your own image with all the necessary tools for building, such as Maven and/or Gradle, the docker command-line tool, plus all sorts of curl, jq, envsubst. In general, here it is:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
ARG JAVA_VERSION="16" FROM bellsoft/liberica-openjdk-alpine-musl:${JAVA_VERSION} # We have to keep this up to date. ARG MAVEN3_VERSION="3.8.1" ARG GRADLE_VERSION="7.1.1" ARG DOCKER_VERSION="20.10.8" # Links to download binary files. ARG MAVEN3_CLI_URL="http://mirror.linux-ia64.org/apache/maven/maven-3/$MAVEN3_VERSION/binaries/apache-maven-$MAVEN3_VERSION-bin.zip" ARG GRADLE_CLI_URL="https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip" ARG DOCKER_CLI_URL="https://download.docker.com/linux/static/stable/x86_64/docker-$DOCKER_VERSION.tgz" # Add installation targets to PATH. ENV PATH="/opt/apache-maven-$MAVEN3_VERSION/bin:/opt/gradle-$GRADLE_VERSION/bin:${PATH}" # Ultimately update OS packages. # hadolint ignore=DL3017, DL3018, DL3019 RUN echo "http://dl-3.alpinelinux.org/alpine/latest-stable/main" > /etc/apk/repositories \ && echo "http://dl-3.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories \ && apk update \ && apk upgrade --available \ && apk add git \ curl \ wget \ tzdata \ gettext \ busybox-extras \ && rm -rf /var/cache/apk/* \ # ===== ===== ===== Install Apache Maven ===== ===== ===== && curl -s -L -o /opt/apache-maven-$MAVEN3_VERSION-bin.zip $MAVEN3_CLI_URL \ && unzip -q -d /opt /opt/apache-maven-$MAVEN3_VERSION-bin.zip \ && rm -f /opt/apache-maven-$MAVEN3_VERSION-bin.zip \ && mvn --version \ && echo " Maven ${MAVEN3_VERSION} installed." \ # ===== ===== ===== Install Gradle ===== ===== ===== && curl -s -L -o /opt/gradle-$GRADLE_VERSION-bin.zip $GRADLE_CLI_URL \ && unzip -q -d /opt /opt/gradle-$GRADLE_VERSION-bin.zip \ && rm -f /opt/gradle-$GRADLE_VERSION-bin.zip \ && gradle --version \ && echo " Gradle ${GRADLE_VERSION} installed." \ # ===== ===== ===== Install Docker CLI ===== ===== ===== && curl -s -L -o /tmp/docker-$DOCKER_VERSION.tgz $DOCKER_CLI_URL \ && tar -x -z -C /tmp -f /tmp/docker-$DOCKER_VERSION.tgz \ && mv /tmp/docker/docker /usr/local/bin/ \ && rm -f /tmp/docker-$DOCKER_VERSION.tgz \ && rm -rf /tmp/docker \ && docker --version \ && echo " Docker CLI ${DOCKER_VERSION} installed." |
The tricky package update came after the demands from the security guards on one of the projects, where scanning of images for vulnerabilities was built into the Container Registry. We noticed that the base image of Liberica JDK is built on the basis of Alpine 3.12 (at the time of this writing), and the current release is 3.14. Therefore, in this way we update the entire OS to the latest version. Before building, all docker files are run through hadolint, and you can see that here we are deliberately breaking several of its rules. We also act in the Dockerfile of the application itself, at the end of the article we will give it as a bonus.
Since almost all projects are closed, you have to drag this Dockerfile into the repository of each project and add a separate job to build it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
variables: TOOLING_IMAGE: $CI_REGISTRY_IMAGE/tooling/java:16 tooling: stage: tooling image: docker:stable script: - cd tooling - docker run --rm -i hadolint/hadolint < Dockerfile - docker build --rm --tag $TOOLING_IMAGE . - docker login --username $CI_REGISTRY_USER --password $CI_REGISTRY_PASSWORD $CI_REGISTRY - docker push $TOOLING_IMAGE only: changes: - tooling/* allow_failure: true |
Further assembly of the application is already carried out using $ TOOLING_IMAGE as the image job.
In fact, another bold component has recently appeared in tooling looks. GitLab is able to display a lot in the Merge Request, for example, it can show a list of run tests and even display the log of each individual one. To do this, you just need to indicate in the job description where GitLab should take JUnit reports in XML format from. This is how it looks:
This feature costs almost nothing. It is much more interesting to add support for Code Coverage to your project in order to see directly in the diff which code is covered by tests and which is not:
To do this, firstly, you need to use JaCoCo during assembly to get a report in its format. For this, the jacoco-maven-plugin has been added to the pom.xml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
<plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.7</version> <configuration> <destFile>jacoco-unit.exec</destFile> <dataFile>jacoco-unit.exec</dataFile> </configuration> <executions> <execution> <id>jacoco-prepare-agent</id> <phase>test-compile</phase> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>jacoco-report</id> <phase>prepare-package</phase> <goals> <goal>report</goal> </goals> </execution> <execution> <id>jacoco-check</id> <phase>package</phase> <goals> <goal>check</goal> </goals> <configuration> <rules> <rule> <element>BUNDLE</element> <limits> <limit> <counter>COMPLEXITY</counter> <value>COVEREDRATIO</value> <minimum>0.40</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> </executions> </plugin> |
Secondly, GitLab can show Code Coverage only in Cobertura format. From the links in the documentation, you can find Python scripts that convert the former to the latter. And yes, you need Python for that. In this case, we will trust the proposed instructions, and this is what we get. Add one more command to the tiling image to install everything you need:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# hadolint ignore=DL3013, DL3018 RUN apk --no-cache add python3 \ python3-dev \ py3-pip \ build-base gcc \ libxml2-dev \ libxslt-dev \ && pip3 install --no-cache-dir --upgrade pip wheel \ && pip3 install --no-cache-dir lxml \ && curl -s -L -o /opt/cover2cover.py https://gitlab.com/haynes/jacoco2cobertura/-/raw/main/cover2cover.py \ && curl -s -L -o /opt/source2filename.py https://gitlab.com/haynes/jacoco2cobertura/-/raw/main/source2filename.py \ && echo " Python scripts for code coverage installed." |
Then we can put everything together in one job in GitLab.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
# Job, with which we collect the application image from the source. my-app: image: $TOOLING_IMAGE variables: # Required for caching dependencies between runs. MAVEN_OPTS: "-Dmaven.repo.local=${CI_PROJECT_DIR}/.m2/repository" MAVEN_CLI_OPTS: "--batch-mode" # GRADLE_USER_HOME: "$CI_PROJECT_DIR/.gradle" # For the convenience of calling commands for converting the Code Coverage format. JACOCO_REPORT: target/site/jacoco/jacoco.xml COBERTURA_XML: target/site/cobertura.xml script: # Compiling the project, running the tests and getting the final .jar. - mvn $MAVEN_CLI_OPTS verify # We transfer the finished file to the directory above, and target is added to .dockerignore. - mv -f target/*.jar application.jar # We validate the Dockerfile, build the application into an image and push it wherever needed. - docker run --rm -i hadolint/hadolint < Dockerfile - docker build --tag my-app . - docker push my-app after_script: # This comment remained here from the GitLab CI documentation. # Report is generated by jacoco-maven-plugin in your pom.xml. # These lines convert it to cobertura format. - python3 /opt/cover2cover.py ${JACOCO_REPORT} src/main/java > ${COBERTURA_XML} - python3 /opt/source2filename.py ${COBERTURA_XML} cache: key: m2-cache paths: - .m2/repository # - .gradle/caches/ # - .gradle/wrapper/ # - .gradle/build-cache/ artifacts: reports: # Save JUnit report on passed unit tests. junit: - '*/target/surefire-reports/TEST-*.xml' - '*/target/failsafe-reports/TEST-*.xml' # Keeping JaCoCO's Code Coverage Report. cobertura: - '*/target/site/cobertura.xml' |
Thus, we achieve our goals. Above they promised to give a version of their Dockerfile for the application. Here it is:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
# Temporary image for unpacking the .jar application. FROM bellsoft/liberica-openjdk-alpine-musl:16 AS builder # Unpack the .jar file. WORKDIR /extracted COPY application.jar /extracted/application.jar RUN java -Djarmode=layertools -jar /extracted/application.jar extract # We start again with a clean slate. Alternatively, here you can replace openjdk # on openjre to reduce the final image, but I decided to rummage through the same layers. FROM bellsoft/liberica-openjdk-alpine-musl:16 # The application will work in the Europe / Moscow time zone. ENV TZ="Europe/Moscow" # Installing package updates and additional tools. # hadolint ignore=DL3017, DL3018 RUN echo "http://dl-3.alpinelinux.org/alpine/latest-stable/main" > /etc/apk/repositories \ && echo "http://dl-3.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories \ && apk update \ && apk upgrade --available \ && rm -rf /var/cache/apk/* \ && apk --no-cache add jq \ zip \ curl \ htop \ tzdata \ busybox-extras \ # Create a user to run the container as non-root. && mkdir -p /my-demo-app \ && chmod 666 /my-demo-app \ && adduser -S -D -h /my-demo-app -u 3456 my-demo-app # The application will run as the user created above. # This is a requirement of the system administrators in charge of Kubernetes. USER my-demo-app WORKDIR /my-demo-app COPY --from=builder /extracted/dependencies/ /my-demo-app/ COPY --from=builder /extracted/snapshot-dependencies/ /my-demo-app/ COPY --from=builder /extracted/spring-boot-loader/ /my-demo-app/ COPY --from=builder /extracted/application/ /my-demo-app/ # REST API. EXPOSE 8090 # Actuator. EXPOSE 8081 ENTRYPOINT ["java", \ # JVM settings. "-XX:MaxDirectMemorySize=50M", \ "-XX:MaxMetaspaceSize=200M", \ "-XX:ReservedCodeCacheSize=120M", \ "-Xss512K", \ "-Xms480M", "-Xmx480M", \ # Look at my horse, my horse is amazing! "-XX:+UseShenandoahGC", \ # Informative NPE. "-XX:+ShowCodeDetailsInExceptionMessages", \ # Allowing Preview Features. "-XX:+UnlockExperimentalVMOptions", \ "--enable-preview", \ # Launching Unpacked Fat Jar. "org.springframework.boot.loader.JarLauncher" \ ] |
All of the above can turn out to be the wildest over-engineering.
In our experience, there is only one Gradle project that we build in GitLab CI, and that one is frozen, so you may have to finish something in the provided material to get a report on unit tests and code coverage.
