Sometimes you need to talk directly to your Kubernetes services. The problem is that the Kubernetes services are over there, floating in that mysterious aether known as the Cloud.

Some projects bring the cloud down to your local development environment, like kubefwd. This works for most reasonable applications, but some applications are less reasonable than others. For those applications, it’s useful to go to the cloud, accessing them the same way you would expect any other service to access them.

Here’s how I set up a remote development environment in my Kubernetes cluster.

Keeping state in a remote pod

The first problem with setting up a development environment is persistence. After all, you can kubectl run --image=debian dev whenever you like, but you’ll have to keep installing the same tools and retyping the same commands every time the pod restarts. Not ideal.

Fortunately, Kubernetes has the concept of PersistentVolumes — volumes mapped to cloud block devices. These have the advantage of surviving pod restarts or cluster scaling events. So to start off, we’ll define a StatefulSet called dev and define a volumeClaimTemplate for it.

---
apiVersion: v1
kind: Namespace
metadata:
  name: dev
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dev
  namespace: dev
spec:
  selector:
    matchLabels:
      app: dev
  serviceName: dev
  volumeClaimTemplates:
    - metadata:
        name: devfs
      spec:
        storageClassName: local  # Replace this with block storage!
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

Now, we could just mount a PersistentVolume to root’s home directory, but it’d be awfully convenient if the entire development environment persisted, including packages and tools installed to the root drive.

Unfortunately, PersistentVolumes can’t be mounted at the root of a container filesystem. No matter — with chroot, we can make any folder we like into our root filesystem. The “host” image can be a simple busybox container that exists only to chroot into our actual development environment. In this case, I’m using Debian, but other distributions’ Docker images will work just as well.

Populating the PersistentVolume

Before we can chroot into our development environment, first we have to seed the PersistentVolume with a root filesystem from an existing image.

To do that, we’ll use Kubernetes’s init containers feature. Init containers run prior to a pod’s main containers and are expected to be short-lived jobs that perform some prep work before the main containers start. In our case, we’ll be using an init container to copy the contents of an entire container image to a PersistentVolume.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dev
  namespace: dev
spec:
  selector:
    matchLabels:
      app: dev
  serviceName: dev
  template:
    metadata:
      labels:
        app: dev
    spec:
      initContainers:
        - name: dev-base
          image: debian:latest  # Your base dev environment
          imagePullPolicy: Always
          command:
            - /bin/sh
            - -c
            - >-
              [ ! -f /mnt/rootfs/usr/bin/env ] && (
                apt-get update &&
                apt-get install -y rsync &&
                rsync -a --exclude /mnt --exclude /sys --exclude /proc / /mnt/rootfs
              ) || true
          volumeMounts:
            - name: devfs
              mountPath: /mnt/rootfs
  volumeClaimTemplates:
    - metadata:
        name: devfs
      spec:
        storageClassName: local  # Replace this with block storage!
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

After a successful run of the init container, the devfs volume will contain a perfect copy of the debian image.

Setting up the host container

With /mnt/rootfs seeded, now it’s time to add the host container. Since we only need it for a chroot, busybox is an obvious and lightweight choice.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dev
  namespace: dev
spec:
  selector:
    matchLabels:
      app: dev
  serviceName: dev
  template:
    metadata:
      labels:
        app: dev
    spec:
      initContainers:
        - name: dev-base
          image: debian:latest  # Your base dev environment
          imagePullPolicy: Always
          command:
            - /bin/sh
            - -c
            - >-
              [ ! -f /mnt/rootfs/usr/bin/env ] && (
                apt-get update &&
                apt-get install -y rsync &&
                rsync -a --exclude /mnt --exclude /sys --exclude /proc / /mnt/rootfs
              ) || true
          volumeMounts:
            - name: devfs
              mountPath: /mnt/rootfs
      containers:
        - name: dev
          image: busybox
          imagePullPolicy: Always
          command: ['/bin/cat']
          tty: true
          volumeMounts:
            - name: devfs
              mountPath: /mnt/rootfs
          securityContext:
            privileged: true
  volumeClaimTemplates:
    - metadata:
        name: devfs
      spec:
        storageClassName: local  # Replace this with block storage!
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

By allocating a psuedo-terminal and running /bin/cat, the host container’s main process blocks forever, waiting for input that will never come. This allows the container to stay running even as we jump into and out of it.

Setting privileged: true lets us use mount commands to share /proc and /dev with the development environment, letting commands like ps aux work as we would expect.

Now we’re finished defining the dev pod, so run kubectl apply -f $your_file_here.

Jumping into the development environment

The remote development environment is up and running; now we just need to get into it!

This alias ensures /proc and /dev are mounted into the chroot, then jumps into the development environment.

alias k8sdev="kubectl exec -n dev -ti pod/dev-0 /bin/sh -- -c '\
    mount -t proc proc /mnt/rootfs/proc 2>/dev/null; \
    mount --rbind dev /mnt/rootfs/dev 2>/dev/null; \
    chroot /mnt/rootfs su -l'"

If all goes well, k8sdev should drop you into root’s home directory. By running ps aux, you should see the indefinitely-blocked /bin/cat as PID 1.

Congratulations! You’ve established a persistent development environment inside your Kubernetes cluster. Now let’s test it out.

Testing persistence

Install a few system packages to get yourself comfortable. Then exit out of the pod and kill it.

kubectl -n dev delete pod/dev-0

Give Kubernetes a minute or two, then run k8sdev again. You should find your system packages safe and sound.

Starting from scratch

If you need to start over, running kubectl delete -f $your_file_here erases everything, including your PersistentVolume. (Be careful!) Then, by running kubectl apply again, it will provision a new volume, and the init container will once again perform a fresh sync of whatever image of your choosing.

Caveats

While the root filesystem will persist, the dev pod itself will not. Don’t run any critical, long-running commands in your dev pod! Kubernetes will always reschedule it if possible, but while the rescheduled pod will restore your files, it won’t restore old processes.

Also, this isn’t a practical way to do heavy development. It uses your Kubernetes cluster’s resources and can always be interrupted. It is, however, excellent for debugging things, especially DNS-sensitive things, in a way that kubectl port-forward or kubefwd won’t allow.

I came up with this scheme while playing around with Kafka and discovering that Kafka brokers really don’t like you to lie about their DNS names. I ended up with a lot of port forwarding commands and a lengthy resolv.conf before throwing my hands in the air and resolving to take the fight to the cluster instead.

All in all, I think it’s a useful tool in my toolbox and a fun abuse of PersistentVolumes.