# Using gradle in docker with Jenkins
Over the last few days I migrated the whole CI/CD pipeline from plain jenkins to docker with jenkins. What sounds like an easy task took way longer that it should have due to various pitfalls when using docker with jenkins. This also applies to builds running in openshift / kubernetes.
In this post I want to share the experience, so you don't have to go through it all again.
# Build process
My primary build tool is gradle. The Project structure I'm using is quite complex but here is a high level overview:
Most of the project can be build using a regular gradle and jdk installation. Only the last modules in the tree require a specific SDK or platform to be build:
# Android
In order to build the android apps the android-sdk is required.
# iOS / OSX
For the iOS / OSX app XCode is required. Since OSX doesn't natively support docker there is no way around using a jenkins agent on a mac.
# Windows
The windows version requires windows for creating the installer. Also docker can't be used here since I'm still using a windows 7 VM for running some backward compatibility tests. Once updated to windows 10 this could be moved into docker as well.
# Target architecture
After analyzing the project it gets clear that we need three different docker containers:
- Regular gradle with jdk
- Gradle with jdk and xcfb for integration UI tests
- Gradle with jdk and android SDK
In order run some steps inside a docker container the jenkins pipeline provides a simple api:
docker.image('gradle:6.3-jdk14'){
sh 'gradle ...'
}
at least in theory...
# Caching in $HOME
Gradle uses a .gradle
caching directory which is usually stored in $HOME
(or the java property user.home
)
But this causes issues since jenkins uses its own UID/GID when running inside the container. In other container solution such as openshift a random UID will be used so this issue applies there as well.
Since the container doesn't know
about this UID/GID gradle tries to create its cache dir in /.gradle
which fails due to missing permissions.
we could use --tmpfs
and mount it to /.gradle
but this won't work on windows which we might need later on.
So we just create our own docker image which includes a cache dir with the appropriate permissions.
You could also choose any other path by adjusting $HOME
/user.home
accordingly.
# JDK + desktop
The second issue was that the gradle image comes with a smaller JDK installation
which does not include some packages, including java.desktop
which is required for the desktop build.
Also the linux dependencies for jlink
and jpackage
are missing as well.
# Gradle dockerfile
Here is my complete jenkins compatible dockerfile with JDK-14 and gradle 6.3.
You can adjust the javaUrl
argument or GRADLE_VERSION
ENV if required.
FROM ubuntu:groovy
ARG javaUrl https://download.java.net/java/GA/jdk14.0.1/664493ef4a6946b186ff29eb326336a2/7/GPL/openjdk-14.0.1_linux-x64_bin.tar.gz
USER root
ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8
ENV JAVA_HOME /opt/java/openjdk
ENV JAVA_URL $javaUrl
ENV GRADLE_VERSION=6.3
ENV GRADLE_USER_HOME /.gradle
ENV GRADLE_HOME=/opt/gradle
ENV PATH=${JAVA_HOME}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl ca-certificates fontconfig locales unzip \
&& echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
&& locale-gen en_US.UTF-8 \
&& rm -rf /var/lib/apt/lists/*
RUN echo Downloading java from ${JAVA_URL} \
&& curl -Lo jdk.tar.gz ${JAVA_URL} \
&& tar xzf jdk.tar.gz \
&& rm jdk.tar.gz \
&& mkdir -p /opt/java \
&& mv jdk-* ${JAVA_HOME}
RUN set -o errexit -o nounset \
&& echo "Downloading Gradle" \
&& curl -Lo gradle.zip https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip \
&& echo "Installing Gradle" \
&& unzip gradle.zip \
&& rm gradle.zip \
&& mv "gradle-${GRADLE_VERSION}" "${GRADLE_HOME}/" \
&& ln --symbolic "${GRADLE_HOME}/bin/gradle" /usr/bin/gradle \
&& echo "Testing Gradle installation" \
&& gradle --version
# .gradle and .android are a cache folders
RUN mkdir -p ${GRADLE_USER_HOME}/caches /.android \
&& chmod -R 777 ${GRADLE_USER_HOME} \
&& chmod 777 /.android
# Required for jlink and jpackage
RUN apt-get update \
&& apt-get -y install binutils fakeroot
# Android image
Creating the android image is now pretty easy. All we need to do is to download
the sdk-manager
, fake the "license accepted" part and download the required packages.
As baseImage
you can use the previously build gradle image.
ARG baseImage
FROM $baseImage
USER root
ENV SDK_URL https://dl.google.com/android/repository/commandlinetools-linux-6200805_latest.zip
ENV ANDROID_HOME /usr/local/android-sdk
ENV ANDROID_SDK_ROOT /usr/local/android-sdk
ENV ANDROID_VERSION 28
ENV ANDROID_BUILD_TOOLS_VERSION 30.0.0-rc2
# .android is a cache folder
RUN mkdir -p $ANDROID_HOME/licenses
RUN cd $ANDROID_HOME \
&& curl -o sdk.zip $SDK_URL \
&& unzip sdk.zip \
&& rm sdk.zip
# Create the license files so the sdkmanager doesn't ask for any confirmation
RUN echo "24333f8a63b6825ea9c5514f83c2829b004d1fee" > "$ANDROID_HOME/licenses/android-sdk-license"
RUN echo "84831b9409646a918e30573bab4c9c91346d8abd" > "$ANDROID_HOME/licenses/android-sdk-preview-license"
# Now download the android SDK stuff
RUN $ANDROID_HOME/tools/bin/sdkmanager --sdk_root=${ANDROID_HOME} --update
RUN $ANDROID_HOME/tools/bin/sdkmanager --sdk_root=${ANDROID_HOME} "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" \
"platforms;android-${ANDROID_VERSION}" \
"platform-tools"
# Making it run in jenkins
So now we got all the images ready and can start building the jenkinsfile.
# Shared dependency cache
To reduce build times gradle allows sharing of the dependency cache.
For this to work you need to copy the .gradle/caches/modulees-2
folder of an existing gradle installation (which has all dependencies already cached)
and set the GRADLE_RO_DEP_CACHE
environment variable to the path of the copy.
In my case the folder structure is:
/opt/jenkins/.gradle/caches/modules-2
/opt/jenkins/.gradle-ro-cache/modules-2
Final docker command in the jenkinsfile.
I explicitly set user.home
to /
since the JDK might otherwise return ?
to the application causing for example the android sdk to
give some warnings.
docker.image('dev-core:gradle-java-14').inside('--mount type=bind,src=/opt/jenkins/.gradle-ro-cache,dst=/.gradle-ro-cache') {
withEnv(['GRADLE_RO_DEP_CACHE=/.gradle-ro-cache', 'JAVA_TOOL_OPTIONS=-Duser.home=/']) {
sh 'gradle..'
}
}
If you don't want to use a shared cache you can simply remove the GRADLE_RO_DEP_CACHE
variable, and the volume mount.
Hope this post could help you. Let me know in the comments if you have any questions.