Let’s say you have an automation tool that patches your servers. What patches the server hosting the automation tool? Is it, as the ancients believed, turtles all the way down?
You could perform some kind of magic rug-pulling to try to get the automation
tool to patch its own server without breaking, but package managers can and
will restart any daemon they see fit partway through the process. Something your
automation depends on, like an agent, or even
sshd, might end up getting
bounced before the upgrade is complete. So we need something a bit more
cron? Going by installations alone, it’s the most popular scheduled
job automation tool. It’s simple, it works, and odds are really good that it’s
already running on your system.
And in fact, automated tools already exist that use
cron or other system
scheduling tools to kick off upgrades.
unattended-upgrades are designed to silently patch your systems in
the background and reboot only when necessary. My gripe with them, though, is
that they’re magical by design — they want to run at random times and behave
more like desktop upgrades than a server patch. Being Linux utilities, there are
of course ways of wrangling them into doing what you want, but at that point
I’ve cast aside most of their features in favor of predictability.
Back to basics, then. What’s wrong with sticking the upgrade command in cron directly?
Well, if you use
yum, the answer is… nothing.
yum upgrade -y will run
without any additional user input, just as you’d expect.
apt-get, on the other
hand, is a different story.
apt-get really wants to be run interactively. Even if you run
update && apt-get dist-upgrade -y,
apt-get will try its hardest to get you to
read mailing lists, configuration updates, and warnings. It does’t want to be
responsible for breaking your system and tries to give the end user all the
information they need to make the right decision at every step in the process.
The thing is, though, if you’re not doing anything funky with your package
management, chances are that
apt-get knows better than you and that you’d like
it to get on with whatever it wants to do. Face it, if you were sitting there
reading interactive prompt after interactive prompt, you’d be taking the default
option every time anyway. Defaults are safe; hopefully, defaults represent the
option the developers judged to be the least likely to brick your system. We
just have to give
apt-get some confidence to pick them for us.
So here is a simple script that lets
apt-get shut up and do what it does best.
#!/bin/sh set -e DEBIAN_FRONTEND=noninteractive PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin apt-get update apt-get \ -o Dpkg::Options::=--force-confold \ -o Dpkg::Options::=--force-confdef \ -y dist-upgrade apt-get autoremove -y shutdown -r 0
Put that somewhere useful like
/usr/local/bin/autopatch, mark it executable
chmod +x /usr/local/bin/autopatch), and then invoke it via
cron during a
time when you’re not likely to be using your automation server.
0 2 * * 6 /usr/local/bin/autopatch 2>&1 >> /var/log/autopatch.log
You might notice that this reboots the server indiscriminately; this originates from my deep-seated mistrust of computers. I don’t care whether the server thinks it needs to reboot. I’d rather reboot it anyway and find out sooner rather than later that something won’t come up from a fresh start.
“But downtime!” you might say. Yes, it does mean the server will go down more than it strictly needs to. But this is an automation server, remember. It shouldn’t be doing anything public-facing! Even if the server goes hard down in the middle of the night, the systems it manages should be stable without further intervention. If that isn’t true, then your infrastructure is a ticking time bomb.
In exchange for a brief weekly interruption, my most sensitive server — the server with privileged access to my other servers and cloud hosting accounts — keeps itself up-to-date and secure.