Quick tip for those of you who have just discovered a pip package you’d like to fix or otherwise contribute to.

sudo pip install -e .

This command installs the pip package in an “editable” way (in fact, -e is the short form of --editable) where you can make live changes to the source code without needing to uninstall and reinstall the package.

The only downside to this method is that there is no corresponding uninstall command, at least not one that actually removes any installed command-line tools from your /usr/local/bin. There’s some discussion on the matter at StackOverflow, but I haven’t found any one-size-fits-all solution yet. However, the benefits outweigh this small annoyance for me.

I’m starting on some new projects soon, projects I would very much like to be stateless and highly available. Partially because I like to think of myself as a grown-up developer now who wants to code things the way they ought to be; mostly because I’m a lazy sysadmin who would rather not have to scramble just because the cloud decided to rain that day.

From a dev standpoint, this means looking at Docker containers, which are cool. From an admin standpoint, this means looking at load balancers, which are not.

A Load of Balancers

There are plenty of cloud-based load balancing options available if you’re willing to pay, like Amazon’s imaginatively-named Application Load Balancer or DigitalOcean’s minimalistically-titled Load Balancer. But I’m not willing to pay for things which fail to impress. All of these cloud “load balancers” boil down to the same basic structure:

  1. A pair of cloud servers with DNS based failover.
  2. Health checks.
  3. Some amount of application-level routing.

That sounds exactly like the sort of thing I’d want a crack at cobbling together myself first. Maybe not to the same depth of features as something like Amazon’s offering, but at least something serviceable.

Is “Passive” Good Enough?

There are a number of load balancing solutions out there, HAProxy probably being the most mature one. At first I dismissed Nginx — a technology I’m already using for other purposes — due to its health check limitations. Namely, open-source Nginx only supports passive health checks.

The idea of sending traffic to potentially bad backend servers did not appeal to me. Intutively, this sounded like dropped traffic and missed requests. It’s passive, after all, and passive has a negative connotation in the world of infrastructure. You don’t want to be passive, you want to be proactive! And besides, Nginx Plus offers an “active” solution, and it costs money, so it must be better.

But I’d never tried it, and something sounding bad is not a good enough reason to avoid it. So I set up a quick test lab and gave it a spin.

Turns out Nginx never missed a beat.

In retrospect, this makes perfect sense. The worst case scenario for a passively or actively checking load balancer is exactly the same. No matter how intelligent your load balancer is, it isn’t clairvoyant. There is always a chance of live traffic being passed onto an node that’s about to become unresponsive, and any traffic routing solution worth its salt knows that and has some kind of transparent retry logic baked in.

Really, the only way the above scenario can be avoided at all times is by broadcasting duplicate traffic to multiple backend servers simultaneously and returning the reply from the winner (à la dnsmasq --all-servers). That’s a lot of overhead that would be better used passing new requests!

Now, this isn’t to say that passive health evaluation is good enough for all use cases. There are real network penalties that get paid, penalties that are minimized in this example since it’s all happening over a local network. And while the worst-case scenario for passive and active health checks is identical, active health checks make that scenario (passing live, non-synthetic traffic to a dead node) less likely. You can also get a lot fancier with active health checks by sending known-good requests and expecting positive HTTP response codes. I don’t particularly see that as the load balancer’s job, but that’s a topic for another time.

This demo proves that open-source Nginx is more than capable enough for my needs. And, more importantly, it serves as reminder to always lab out your assumptions. Chances are you’ll learn something.

WebM is probably the future, but like all things containing the word “web,” there’s an element of nuance and surprise that emerges while working with it.

VP9 is supported pretty much everywhere that matters — even Edge manages to produce a video UI, although one imagines it grumbling as it does so — but the only pixel format that actually works across computers and phones alike is YUV420p. For the most part, this detail manifests itself as an additional flag you must pass to your video encoder of choice (e.g. ffmpeg -pix_fmt yuv420p) lest you present a perfectly playable but nevertheless blank video to some, if not all, of your audience.

But there’s another layer here, one that picks apart the strange name of YUV420p, buried in a drawer somewhere with the label “Chroma Subsampling.” For efficiency reasons, YUV420p will keep only one pixel of color information for every four pixels of video, which means a loss of detail even if you tell it to encode “losslessly.”

For the most part, this produces a perfectly acceptable result, unless you’re doing something that lends itself to pixel precision like screencasting a small terminal window (and, let’s be honest, cultivating a habit of getting hung up on details). The correct solution to this is to move to YUV444p, which keeps a good old-fashioned one-to-one ratio of color data to pixels. Except, like so many “standards,” it produces inconsistent results across browsers. In my limited testing, Chrome and Firefox on Android were able to play YUV444p, along with Firefox on Linux, but Chrome, Firefox, and Edge on Windows didn’t have a clue what was going on and put up a black box instead. Even my file browser wasn’t happy with it, if I’m honest, as it generated a sickly-green thumbnail for the video file.

Options for platform-agnostic pixel-precision are slim, it appears, at least if you’re keeping to strictly HTML5 video. I’ve found a simple way around this for now, though. Since chroma subsampling wants to reduce every 2x2 block of pixels to a single chromatic data point, why not feed it 4x the amount of pixels? And to prevent the filesize from growing unnecessarily, don’t let ffmpeg attempt to do any kind of interpolation when scaling up — stick with the nearest neighbor algorithm and make plain ol’ 2x2 chunks.

This is the best I’ve come up with for now, but I’m hoping YUV444p support will be more of a thing in the future to avoid this kind of workaround.

If you’ve tried updating suckless.org’s Simple Term, st, and run into this error:

In file included from x.c:61:
config.h:180:42: error: ‘iso14755’ undeclared here (not in a function)
  { TERMMOD,              XK_I,           iso14755,       {.i =  0} },
                                          ^~~~~~~~

make: *** [Makefile:22: x.o] Error 1

You’re probably working off of a config.h that you generated when ISO 14755 mode was a feature in st, which was removed in 67d0cb6. Remove the line indicated in the error message from your config.h and make should run without further issue.

I have a few deeply held beliefs when it comes to software.

  • If you do something more than twice, it should be automated.
  • It’s always DNS.
  • Build your code in containers.

Even AWS and I seem to agree on that last point. It was a significant contributing factor when choosing Concourse as my personal CI solution. Installing dependencies directly on build nodes is needless snowflakery.

It shouldn’t be your build system’s job to have the right versions of the right resources installed on the right nodes. Docker has already solved the problem of creating well-defined and consistent environments, so use it. Your build server should only know two things: how to run your build software and how to run containers. Everything else is detail better left to build definitions.

The Plan

I’m making the transition from Concourse to Jenkins, which has adequate Docker support in its Declarative Pipelines. Naturally, this is something I’m looking to make extensive use of. But I’d also like to keep Jenkins as self-contained as possible. I want to be able to swap it out for a new version with as little effort as I can. If it’s going haywire, if I need more resources elsewhere, I want to be able to shut down the build system all at once without needing to clean up first. So of course, I want to run it in a container.

But there’s a catch. That Jenkins inside the container will itself need to start containers. How can Docker run Docker?

The Docker Ouroboros

As it turns out, Docker can be convinced to eat its own tail in a number of different ways. There is DooD — Docker outside of Docker — where the Docker binary runs inside the container, but uses the host’s unix socket to communicate with the host Docker. There’s also DinD — Docker in Docker — where Docker runs completely inside the container, which only requires the --privileged flag to work.

I don’t want to use DooD, not only because the acronym is sillier, but also because it means a shutdown of the Jenkins container might still leave unnecessary containers running on the host. I also had a surprising amount of difficulty when testing it locally on a Mac. Something about permission issues, which could only be resolved by giving things more permissions than I’d like

So that leaves DinD. Except Jenkins doesn’t like DinD.

The “right” way to use DinD is to treat Docker itself as just another service, and in container-land that means it should run a separate container. Expose Docker over a port, point the Jenkins container to that port, and… mysterious errors referencing /var/lib/docker. Jenkins wants Docker running on the same system. Fine then.

Introducing JenkDIND

At this point, if I were setting up a serious Jenkins build system, I would probably stop and use DooD or, better yet, just install Jenkins and Docker directly on the Jenkins slaves. But I’m not setting a serious build system. I’m not an organization, I’m just a guy setting up a single Jenkins master for his personal projects. I’m willing to sacrifice a little bit of “doing things the right way” if I’m gaining a lot of convenience.

Plus, I like breaking stuff.

With a little help from s6, I created JenkDIND: a multi-process Docker image with Jenkins and Docker running side-by-side in a manner as near to a “real” installation as possible. Jenkins runs as uid 1000 with docker but no sudo privileges, while Docker runs root and does not reach out to the host Docker to do its job.

FROM jenkins/jenkins:lts-alpine

USER root

ADD https://github.com/just-containers/s6-overlay/releases/download\
/v1.21.7.0/s6-overlay-amd64.tar.gz \
/tmp/

RUN apk add --no-cache docker shadow \
 && gpasswd docker -a jenkins

RUN mkdir -p /etc/services.d/docker /etc/services.d/jenkins \
 && printf "#!/usr/bin/execlineb -P\n"\
    "dockerd" > /etc/services.d/docker/run \
 && printf "#!/usr/bin/execlineb -P\n"\
    "with-contenv\n"\
    "s6-setuidgid jenkins\n"\
    "s6-env HOME=/var/jenkins_home\n"\
    "/usr/local/bin/jenkins.sh" > /etc/services.d/jenkins/run \
 && gunzip -c /tmp/s6-overlay-amd64.tar.gz | tar -xf - -C /

ENTRYPOINT ["/init"]

JenkDIND is recommended to be run with two volumes: a volume mounted at /var/jenkins_home to keep your Jenkins configuration and state in, and another volume mounted at /var/lib/docker to keep Docker’s stuff in. Clearing the Docker cache is simple, stop the container and docker volume rm the entire Docker volume. (Or is that Docker Docker volume?)

docker run \
  --name jenkins \
  -e TZ=America/New_York \
  -p 8080:8080 \
  -v jenkins_home:/var/jenkins_home \
  -v docker_store:/var/lib/docker \
  --privileged \
  -d \
  awkspace/jenkdind

JenkDIND is available on GitHub.