I have a few deeply held beliefs when it comes to software.
- If you do something more than twice, it should be automated.
- It’s always DNS.
- Build your code in containers.
Even AWS and I seem to agree on that last
It was a significant contributing factor when choosing
Concourse as my personal CI solution. Installing
dependencies directly on build nodes is needless
It shouldn’t be your build system’s job to have the right versions of the right
resources installed on the right nodes. Docker has already solved the problem of
creating well-defined and consistent environments, so use it. Your build server
should only know two things: how to run your build software and how to run
containers. Everything else is detail better left to build definitions.
I’m making the transition from Concourse to Jenkins,
which has adequate Docker support in its Declarative
Pipelines. Naturally, this is
something I’m looking to make extensive use of. But I’d also like to keep
Jenkins as self-contained as possible. I want to be able to swap it out for a
new version with as little effort as I can. If it’s going haywire, if I need
more resources elsewhere, I want to be able to shut down the build system all at
once without needing to clean up first. So of course, I want to run it in a
But there’s a catch. That Jenkins inside the container will itself need to
start containers. How can Docker run Docker?
The Docker Ouroboros
As it turns out, Docker can be convinced to eat its own tail in a number of
different ways. There is DooD — Docker outside of Docker — where the Docker
binary runs inside the container, but uses the host’s unix socket to communicate
with the host Docker. There’s also DinD — Docker in Docker — where Docker runs
completely inside the container, which only requires the
--privileged flag to
I don’t want to use DooD, not only because the acronym is sillier, but also
because it means a shutdown of the Jenkins container might still leave
unnecessary containers running on the host. I also had a surprising amount of
difficulty when testing it locally on a Mac. Something about permission
which could only be resolved by giving things more permissions than I’d like
So that leaves DinD. Except Jenkins doesn’t like DinD.
The “right” way to use DinD is to treat Docker itself as just another service,
and in container-land that means it should run a separate container. Expose
Docker over a port, point the Jenkins container to that port, and… mysterious
/var/lib/docker. Jenkins wants Docker running on the same
system. Fine then.
At this point, if I were setting up a serious Jenkins build system, I would
probably stop and use DooD or, better yet, just install Jenkins and Docker
directly on the Jenkins slaves. But I’m not setting a serious build system. I’m
not an organization, I’m just a guy setting up a single Jenkins master for his
personal projects. I’m willing to sacrifice a little bit of “doing things the
right way” if I’m gaining a lot of convenience.
Plus, I like breaking stuff.
With a little help from s6, I
created JenkDIND: a multi-process Docker image with Jenkins and Docker running
side-by-side in a manner as near to a “real” installation as possible. Jenkins
runs as uid
docker but no
sudo privileges, while Docker runs
root and does not reach out to the host Docker to do its job.
RUN apk add --no-cache docker shadow \
&& gpasswd docker -a jenkins
RUN mkdir -p /etc/services.d/docker /etc/services.d/jenkins \
&& printf "#!/usr/bin/execlineb -P\n"\
"dockerd" > /etc/services.d/docker/run \
&& printf "#!/usr/bin/execlineb -P\n"\
"/usr/local/bin/jenkins.sh" > /etc/services.d/jenkins/run \
&& gunzip -c /tmp/s6-overlay-amd64.tar.gz | tar -xf - -C /
JenkDIND is recommended to be run with two volumes: a volume mounted at
/var/jenkins_home to keep your Jenkins configuration and state in, and another
volume mounted at
/var/lib/docker to keep Docker’s stuff in. Clearing the
Docker cache is simple, stop the container and
docker volume rm the entire
Docker volume. (Or is that Docker Docker volume?)
docker run \
--name jenkins \
-e TZ=America/New_York \
-p 8080:8080 \
-v jenkins_home:/var/jenkins_home \
-v docker_store:/var/lib/docker \
JenkDIND is available on GitHub.