If you’ve tried updating suckless.org’s Simple Term, st, and run into this error:

In file included from x.c:61:
config.h:180:42: error: ‘iso14755’ undeclared here (not in a function)
  { TERMMOD,              XK_I,           iso14755,       {.i =  0} },
                                          ^~~~~~~~

make: *** [Makefile:22: x.o] Error 1

You’re probably working off of a config.h that you generated when ISO 14755 mode was a feature in st, which was removed in 67d0cb6. Remove the line indicated in the error message from your config.h and make should run without further issue.

I have a few deeply held beliefs when it comes to software.

  • If you do something more than twice, it should be automated.
  • It’s always DNS.
  • Build your code in containers.

Even AWS and I seem to agree on that last point. It was a significant contributing factor when choosing Concourse as my personal CI solution. Installing dependencies directly on build nodes is needless snowflakery.

It shouldn’t be your build system’s job to have the right versions of the right resources installed on the right nodes. Docker has already solved the problem of creating well-defined and consistent environments, so use it. Your build server should only know two things: how to run your build software and how to run containers. Everything else is detail better left to build definitions.

The Plan

I’m making the transition from Concourse to Jenkins, which has adequate Docker support in its Declarative Pipelines. Naturally, this is something I’m looking to make extensive use of. But I’d also like to keep Jenkins as self-contained as possible. I want to be able to swap it out for a new version with as little effort as I can. If it’s going haywire, if I need more resources elsewhere, I want to be able to shut down the build system all at once without needing to clean up first. So of course, I want to run it in a container.

But there’s a catch. That Jenkins inside the container will itself need to start containers. How can Docker run Docker?

The Docker Ouroboros

As it turns out, Docker can be convinced to eat its own tail in a number of different ways. There is DooD — Docker outside of Docker — where the Docker binary runs inside the container, but uses the host’s unix socket to communicate with the host Docker. There’s also DinD — Docker in Docker — where Docker runs completely inside the container, which only requires the --privileged flag to work.

I don’t want to use DooD, not only because the acronym is sillier, but also because it means a shutdown of the Jenkins container might still leave unnecessary containers running on the host. I also had a surprising amount of difficulty when testing it locally on a Mac. Something about permission issues, which could only be resolved by giving things more permissions than I’d like

So that leaves DinD. Except Jenkins doesn’t like DinD.

The “right” way to use DinD is to treat Docker itself as just another service, and in container-land that means it should run a separate container. Expose Docker over a port, point the Jenkins container to that port, and… mysterious errors referencing /var/lib/docker. Jenkins wants Docker running on the same system. Fine then.

Introducing JenkDIND

At this point, if I were setting up a serious Jenkins build system, I would probably stop and use DooD or, better yet, just install Jenkins and Docker directly on the Jenkins slaves. But I’m not setting a serious build system. I’m not an organization, I’m just a guy setting up a single Jenkins master for his personal projects. I’m willing to sacrifice a little bit of “doing things the right way” if I’m gaining a lot of convenience.

Plus, I like breaking stuff.

With a little help from s6, I created JenkDIND: a multi-process Docker image with Jenkins and Docker running side-by-side in a manner as near to a “real” installation as possible. Jenkins runs as uid 1000 with docker but no sudo privileges, while Docker runs root and does not reach out to the host Docker to do its job.

FROM jenkins/jenkins:lts-alpine

USER root

ADD https://github.com/just-containers/s6-overlay/releases/download\
/v1.21.7.0/s6-overlay-amd64.tar.gz \
/tmp/

RUN apk add --no-cache docker shadow \
 && gpasswd docker -a jenkins

RUN mkdir -p /etc/services.d/docker /etc/services.d/jenkins \
 && printf "#!/usr/bin/execlineb -P\n"\
    "dockerd" > /etc/services.d/docker/run \
 && printf "#!/usr/bin/execlineb -P\n"\
    "with-contenv\n"\
    "s6-setuidgid jenkins\n"\
    "s6-env HOME=/var/jenkins_home\n"\
    "/usr/local/bin/jenkins.sh" > /etc/services.d/jenkins/run \
 && gunzip -c /tmp/s6-overlay-amd64.tar.gz | tar -xf - -C /

ENTRYPOINT ["/init"]

JenkDIND is recommended to be run with two volumes: a volume mounted at /var/jenkins_home to keep your Jenkins configuration and state in, and another volume mounted at /var/lib/docker to keep Docker’s stuff in. Clearing the Docker cache is simple, stop the container and docker volume rm the entire Docker volume. (Or is that Docker Docker volume?)

docker run \
  --name jenkins \
  -e TZ=America/New_York \
  -p 8080:8080 \
  -v jenkins_home:/var/jenkins_home \
  -v docker_store:/var/lib/docker \
  --privileged \
  -d \
  awkspace/jenkdind

JenkDIND is available on GitHub.

After spending some time working in AWS Cloudformation and running cfn-lint manually, I decided to incorporate it into my Emacs config.

Here’s a basic cloudformation-mode for DOOM Emacs, complete with a flycheck checker for cfn-lint:

;; cloudformation-mode

(define-derived-mode cfn-mode yaml-mode
  "Cloudformation"
  "Cloudformation template mode.")

(add-to-list 'auto-mode-alist '(".template\\'" . cfn-mode))
(after! flycheck
  (flycheck-define-checker cfn-lint
    "A Cloudformation linter using cfn-python-lint.

See URL 'https://github.com/awslabs/cfn-python-lint'."
    :command ("cfn-lint" "-f" "parseable" source)
    :error-patterns (
                     (warning line-start (file-name) ":" line ":" column
                              ":" (one-or-more digit) ":" (one-or-more digit) ":"
                              (id "W" (one-or-more digit)) ":" (message) line-end)
                     (error line-start (file-name) ":" line ":" column
                            ":" (one-or-more digit) ":" (one-or-more digit) ":"
                            (id "E" (one-or-more digit)) ":" (message) line-end)
                     )
    :modes (cfn-mode)
    )
    (add-to-list 'flycheck-checkers 'cfn-lint)
  )

Of course, this only covers the YAML case, and technically JSON is valid Cloudformation as well… but who hates themselves enough to write Cloudformation in JSON?

inotifywait is a handy utility that listens for filesystem events you specify and prints them to standard out. This is particularly useful if, say, you want to automatically redeploy a website whenever something changes.

(Jekyll comes with this functionality out-of-the-box, but if you have more than one site, or other non-Jekyll content that changes, it may be desirable to deploy everything together.)

The naive approach to this problem might look something like:

# Filesystem events specific to file creation or modification
events="modify,attrib,close_write,move,create,delete"

# -m = monitor (don’t quit after the first event)
# -r = recursive
inotifywait -m -r -e "$events" $site_dir | while read ; do deploy_site ; done

But the problem with filesystem events is that lots of them can fire simultaneously. A single file save produces multiple events, and a project-wide find/replace opens the floodgates. Since the site will deploy on every individual event, even minor changes kick off 3-4 rebuilds in a row. That’s pretty silly.

I found a few patterns out on the ’net to batch inotifywait messages, but most of them were pretty complicated. I wondered if it might be possible to do the same thing in a more straightforward fashion, and came up with this:

echo "Watching source directory for changes."

events="modify,attrib,close_write,move,create,delete"

while true
do

    count=0

    while read -t 1
    do
        (( count++ ))
    done

    if [[ $count -gt 0 ]]
    then
        echo "$count changes detected."
        deploy_site
    fi

done < <(inotifywait -m -r -e "$events" --exclude '/\..+' $dir 2>/dev/null)

As it turns out, read accepts a timeout value! Now any inotify events less than a second apart will be counted as part of the same batch and will result in only a single deploy_site call.

A word of warning: failure to exclude .git folders will result in a ton of filesystem events. In the above code, I ignore all dotfiles (and folders) to avoid this problem.

I’d been looking for some space on the internet where I could put some words for others to see. Then I decided to make my own space instead.

awk.space is written in Emacs and powered by Jekyll and Amazon S3. Plus some of my own magic that I’ll probably talk about in future blog posts.

The site is split into two different spaces: fiction-writing and blog-writing. Or fiction and nonfiction, if you like. Like a library, except not as cool. Or instagramable.

There’s not much here at the moment, but I hope you enjoy your visit anyway.