awkspace

Jenkins in a Box

awk -

I have a few deeply held beliefs when it comes to software.

  • If you do something more than twice, it should be automated.
  • It’s always DNS.
  • Build your code in containers.

Even AWS and I seem to agree on that last point. It was a significant contributing factor when choosing Concourse as my personal CI solution. Installing dependencies directly on build nodes is needless snowflakery.

It shouldn’t be your build system’s job to have the right versions of the right resources installed on the right nodes. Docker has already solved the problem of creating well-defined and consistent environments, so use it. Your build server should only know two things: how to run your build software and how to run containers. Everything else is detail better left to build definitions.

The Plan

I’m making the transition from Concourse to Jenkins, which has adequate Docker support in its Declarative Pipelines. Naturally, this is something I’m looking to make extensive use of. But I’d also like to keep Jenkins as self-contained as possible. I want to be able to swap it out for a new version with as little effort as I can. If it’s going haywire, if I need more resources elsewhere, I want to be able to shut down the build system all at once without needing to clean up first. So of course, I want to run it in a container.

But there’s a catch. That Jenkins inside the container will itself need to start containers. How can Docker run Docker?

The Docker Ouroboros

As it turns out, Docker can be convinced to eat its own tail in a number of different ways. There is DooD — Docker outside of Docker — where the Docker binary runs inside the container, but uses the host’s unix socket to communicate with the host Docker. There’s also DinD — Docker in Docker — where Docker runs completely inside the container, which only requires the --privileged flag to work.

I don’t want to use DooD, not only because the acronym is sillier, but also because it means a shutdown of the Jenkins container might still leave unnecessary containers running on the host. I also had a surprising amount of difficulty when testing it locally on a Mac. Something about permission issues, which could only be resolved by giving things more permissions than I’d like

So that leaves DinD. Except Jenkins doesn’t like DinD.

The “right” way to use DinD is to treat Docker itself as just another service, and in container-land that means it should run a separate container. Expose Docker over a port, point the Jenkins container to that port, and... mysterious errors referencing /var/lib/docker. Jenkins wants Docker running on the same system. Fine then.

Introducing JenkDIND

At this point, if I were setting up a serious Jenkins build system, I would probably stop and use DooD or, better yet, just install Jenkins and Docker directly on the Jenkins slaves. But I’m not setting a serious build system. I’m not an organization, I’m just a guy setting up a single Jenkins master for his personal projects. I’m willing to sacrifice a little bit of “doing things the right way” if I’m gaining a lot of convenience.

Plus, I like breaking stuff.

With a little help from s6, I created JenkDIND: a multi-process Docker image with Jenkins and Docker running side-by-side in a manner as near to a “real” installation as possible. Jenkins runs as uid 1000 with docker but no sudo privileges, while Docker runs root and does not reach out to the host Docker to do its job.

FROM jenkins/jenkins:lts-alpine

USER root

ADD https://github.com/just-containers/s6-overlay/releases/download\
/v1.21.7.0/s6-overlay-amd64.tar.gz \
/tmp/

RUN apk add --no-cache docker shadow \
 && gpasswd docker -a jenkins

RUN mkdir -p /etc/services.d/docker /etc/services.d/jenkins \
 && printf "#!/usr/bin/execlineb -P\n"\
    "dockerd" > /etc/services.d/docker/run \
 && printf "#!/usr/bin/execlineb -P\n"\
    "with-contenv\n"\
    "s6-setuidgid jenkins\n"\
    "s6-env HOME=/var/jenkins_home\n"\
    "/usr/local/bin/jenkins.sh" > /etc/services.d/jenkins/run \
 && gunzip -c /tmp/s6-overlay-amd64.tar.gz | tar -xf - -C /

ENTRYPOINT ["/init"]

JenkDIND is recommended to be run with two volumes: a volume mounted at /var/jenkins_home to keep your Jenkins configuration and state in, and another volume mounted at /var/lib/docker to keep Docker’s stuff in. Clearing the Docker cache is simple, stop the container and docker volume rm the entire Docker volume. (Or is that Docker Docker volume?)

docker run \
  --name jenkins \
  -e TZ=America/New_York \
  -p 8080:8080 \
  -v jenkins_home:/var/jenkins_home \
  -v docker_store:/var/lib/docker \
  --privileged \
  -d \
  awkspace/jenkdind

JenkDIND is available on GitHub.