So you’ve spun up a fresh Linux Workspace and discovered in short order that it is ugly as sin. Fret not, this is as configurable a Linux machine as any — but in different ways than you might be used to.

Enable SSH

Fortunately, Linux WorkSpaces run sshd by default. However, there’s no way to get to it from the Internet.

This is worth fixing up sooner rather than later, as you may run into a problem in the future with your workspace becoming Unhealthy. An Unhealthy WorkSpace is a WorkSpace that AWS can’t reach, usually caused by networking problems. You won’t be able to connect to an Unhealthy WorkSpace through the WorkSpaces client, so it’s good to have SSH handy for debugging.

Before exposing port 22 to the Internet, edit /etc/ssh/sshd_config and tighten up the security a little bit: set ChallengeResponseAuthentication to no. This prevents password logins and allows only key-based authentication. An exposed SSH port in an AWS IP block is a massive target for brute-force attacks; no sense in letting anyone try.

Now, sudo systemctl restart sshd. Verify it’s running. Add your public key(s) to ~/.ssh/authorized_keys. Once you’ve done that, open up the AWS Console.

The WorkSpace itself isn’t visible in the EC2 console, but the network interface attached to it is. Visit “Network Interfaces” and select the ENI “Created by Amazon WorkSpaces.” While you’re here, by the way, I highly recommend registering the Elastic IP attached to your WorkSpace ENI with some kind of DNS. It’s a lot more convenient to access later, and the Elastic IP won’t change.

Edit the ENI’s security group. Add TCP port 22 from “Anywhere” and save. Your WorkSpace is now SSHable! tail -f /var/log/secure for a good laugh.

Note that if you’re SSHing from a shell, you’ll need to escape the backslash that separates your AD domain from your username, i.e. CORP\\bob rather than CORP\bob.

Disable sudo password prompts

What’s the point of authenticating twice with the same password? Edit /etc/sudoers.d/01-ws-admin-user and change the final ALL to NOPASSWD:ALL.

Enable EPEL

Nope, yum install epel-release doesn’t work with Amazon’s repositories. You’ll need to install the package from the Internet directly.

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Pull in CentOS repositories

The easiest way to do this is to install docker. Don’t forget to add yourself to the Docker group with usermod -a -G docker $USER. Then copy the CentOS repository files to your WorkSpace.

centos=$(docker run centos:7)
sudo docker cp $centos:/etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo
sudo sed -i 's/\$releasever/7/g' /etc/yum.repos.d/CentOS-Base.repo
sudo sed -i 's/\]$/\]\nenabled=0/' /etc/yum.repos.d/CentOS-Base.repo
sudo docker cp $centos:/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 
docker rm -f $centos

Some explanation: $releasever of Amazon Linux is 2, but the compatible CentOS repositories are CentOS 7. So $releasever needs to be hardcoded to 7 in CentOS-Base.repo in order to work.

We also leave these repositories disabled by default to minimize conflicts with Amazon’s own packages. Use them only when you need them by adding --enablerepo= when installing a package, i.e. yum install --enablerepo=extras. Some packages require more obscure dependencies than others, so it’s worth having the CentOS repositories handy.

Change your desktop environment

Personally, I find adding xfce and the Numix theme works wonders for making your WorkSpace more pleasing to the eye with very little effort. If you do run xfce with a custom GTK theme like I do, don’t forget to set the theme style under both Settings Manager > Appearance > Style and Settings Manager > Window Manager > Style > Theme.

sudo yum groupinstall -y xfce
sudo yum install -y numix-gtk-theme numix-icon-theme-circle

To switch window managers, edit /etc/pcoip-agent/pcoip-agent.conf and change pcoip.desktop_session from mate to xfce. If you’ve installed a different desktop environment and are wondering what keyword belongs here, check /usr/share/xsessions:

ls /usr/share/xsessions/*.desktop # List available desktop sessions.

Reboot your WorkSpace and you’ll be in your preferred desktop session.

Change your shell

chsh doesn’t work in this realm because of Active Directory. Instead, add yourself to /etc/passwd and modify your shell the good old-fashioned way.

getent passwd $USER | sudo tee -a /etc/passwd

Then edit /etc/passwd and change your shell to your liking.

Set a monospace font

This isn’t really WorkSpace specific, but I came across this issue while configuring my WorkSpace, so here it is anyway.

Not all desktop environments expose the ability to modify your system’s monospace font. In case you have this issue too, create a ~/.fonts.conf with the following XML format, replacing Terminus with the monospace font of your choice.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<match target="pattern">
  <test qual="any" name="family">
    <string>monospace</string>
  </test>
  <edit name="family" mode="assign">
    <string>Terminus</string>
  </edit>
</match>
</fontconfig>

Why not just use EC2?

Your WorkSpace should now be every bit as comfy as a local VM. But you might be wondering why anyone would bother going through all this special configuration when you could just run your own EC2 instance.

The answer is simple: the WorkSpace is cheaper and Just Works. I run the cheapest Amazon WorkSpace, which is 1 vCPU, 2 GB of RAM, 80 GB of root storage and 10 GB of user storage. It runs me $21 a month. The equivalent EC2/EBS storage is $24.23 a month. Why bother?

As for it Just Working, I don’t have to bother with tuning or securing VNC. The WorkSpaces client is fast, secure, and runs on everything — even Linux, if you install it under WINE. The VirtualBox-like automatic desktop resizing is nice too, which is something even NoMachine struggles to pull off. Having messed with other remote clients in the past, I find the WorkSpaces client to be a more seamless and consistent user experience.

With that, hopefully your WorkSpace is set up the way you like it. Happy hacking!

Amazon WorkSpaces are virtual desktops in the cloud. As designed, they’re an office sysadmin’s dream: a simple thin client, integration with Active Directory, and the ability to rebuild WorkSpaces from scratch with the click of a button. But how are they as a personal desktop? I decided to find out.

They can be incredibly useful

A WorkSpace is valuable so long as you 1. switch computers often and 2. you have actual work to do. If you’re not a hobbyist developer, chances are a cloud drive service will suit your needs just fine.

But if you are a hobbyist devops kinda person, you will know the nightmare of keeping everything “in sync.” Notes are easy: there’s Evernote, Google Keep, Dropbox, good old-fashioned git; code can be committed to private repositories while in progress. But then there’s all the little things. Shell and IDE history. Bookmarks and tabs. Configuration files that are never up to date. Local webservers that you forget to start after every boot.

It’s never anything major, and it’s certainly nothing that stops you from getting stuff done, just a thousand niggling things that aren’t quite the way you expect them to be in the moment. And yes, all of these things are syncable, but most require individual solutions with varying degrees of robustness and headache.

And even when everything is running smoothly, who among us hasn’t had that oh crap! moment of kicking off a long-running script just before the Internet drops?

The brilliance of a WorkSpace can be summed up with the word continuity. You are always exactly where you left off. Your WiFi or your battery may have betrayed you, but your WorkSpace is eternal, ready for you to reconnect and get on with shit.

They need Active Directory

Yes, even the Linux WorkSpaces. It’s just how authentication is done. If you create a WorkSpace manually in the console, AWS will quietly make you a pleasant-sounding Simple AD domain called CORP.

This might ring some alarm bells. After all, nothing in the cloud is free! Except, as it turns out, this is.

As AWS explains on the directory pricing page:

If you use Amazon WorkSpaces, Amazon WorkDocs, or Amazon WorkMail in conjunction with AWS Directory Service, you will not be charged an additional fee for either Simple AD or AD Connector directories registered with these services, as long as you have active users of Amazon WorkSpaces, Amazon WorkDocs, or Amazon WorkMail.

Specifically, you need one active user to keep your small directory free. That’s you!

Since you’re going to have a domain created one way or another, it’s probably worth creating the domain yourself to avoid being stuck with a dreary, generic CORP. You can create your own domain from the ‘Directory Service’ console. Then, once you’ve made it, go to the WorkSpaces console and deploy a WorkSpace from your new domain.

Hourly WorkSpaces are a scam

As you peruse the pricing page for WorkSpaces, you might be tempted to choose an Hourly WorkSpace over an AlwaysOn one. This is because most things in the cloud are cheaper if they’re not on 24/7, and you’re probably not going to be logged in 24 hours a day anyway.

Don’t do it. The per-hour pricing is over five times more expensive going with Hourly over AlwaysOn. Even at a modest use of three hours per day, the cheapest Hourly Linux instance will run you more than the AlwaysOn monthly fee. You also get charged the initial flat Hourly fee immediately, before your first hour ticks over, and you will get charged it again if you re-provision your WorkSpace without waiting for the previous WorkSpace to be deleted. Trust me, I speak from experience.

That’s to say nothing of the awful wait for your Hourly WorkSpace to dredge itself out from whatever depths of the cloud it slumbers in, should you dare to rouse it. Hope you like staring at the word RESUMING with increasing incredulity. Just don’t bother with Hourly.

They will get rebooted

Part of my motivation to check out WorkSpaces was my impatience with Windows clobbering my Linux VMs I worked in whenever it felt like updating. Unfortunately, you don’t escape updates here either. Fortunately, they are completely predictable!

WorkSpace maintenance windows are between midnight and 4am Sunday in the WorkSpace’s timezone. I will happily take a predictable reboot period over the surprises of Windows 10. It would be nice if we could define the maintenance period ourselves, but at least AWS picked a time when people are the least likely to be awake, let alone online.

They are free tier eligible

If you still qualify, you can nab a decent sized WorkSpace under the free tier plan. Go check it out!

As for me, I haven’t been free tier for some time, but I’m finding my $21/month Linux WorkSpace to be fast enough for my needs.

A WorkSpace of my own

While they’re not for everyone, I can report that I’ll probably be staying with my WorkSpace for the immediate future. It’s been an easy transition from having multiple Linux VMs playing configuration leapfrog to a single Linux VM running on someone else’s computer. Having a persistent desktop brings back a certain comfort in a world of increasingly transient devices.

Quick tip for those of you who have just discovered a pip package you’d like to fix or otherwise contribute to.

sudo pip install -e .

This command installs the pip package in an “editable” way (in fact, -e is the short form of --editable) where you can make live changes to the source code without needing to uninstall and reinstall the package.

The only downside to this method is that there is no corresponding uninstall command, at least not one that actually removes any installed command-line tools from your /usr/local/bin. There’s some discussion on the matter at StackOverflow, but I haven’t found any one-size-fits-all solution yet. However, the benefits outweigh this small annoyance for me.

I’m starting on some new projects soon, projects I would very much like to be stateless and highly available. Partially because I like to think of myself as a grown-up developer now who wants to code things the way they ought to be; mostly because I’m a lazy sysadmin who would rather not have to scramble just because the cloud decided to rain that day.

From a dev standpoint, this means looking at Docker containers, which are cool. From an admin standpoint, this means looking at load balancers, which are not.

A Load of Balancers

There are plenty of cloud-based load balancing options available if you’re willing to pay, like Amazon’s imaginatively-named Application Load Balancer or DigitalOcean’s minimalistically-titled Load Balancer. But I’m not willing to pay for things which fail to impress. All of these cloud “load balancers” boil down to the same basic structure:

1. A pair of cloud servers with DNS based failover.
2. Health checks.
3. Some amount of application-level routing.

That sounds exactly like the sort of thing I’d want a crack at cobbling together myself first. Maybe not to the same depth of features as something like Amazon’s offering, but at least something serviceable.

Is “Passive” Good Enough?

There are a number of load balancing solutions out there, HAProxy probably being the most mature one. At first I dismissed Nginx — a technology I’m already using for other purposes — due to its health check limitations. Namely, open-source Nginx only supports passive health checks.

The idea of sending traffic to potentially bad backend servers did not appeal to me. Intutively, this sounded like dropped traffic and missed requests. It’s passive, after all, and passive has a negative connotation in the world of infrastructure. You don’t want to be passive, you want to be proactive! And besides, Nginx Plus offers an “active” solution, and it costs money, so it must be better.

But I’d never tried it, and something sounding bad is not a good enough reason to avoid it. So I set up a quick test lab and gave it a spin.

Turns out Nginx never missed a beat.

In retrospect, this makes perfect sense. The worst case scenario for a passively or actively checking load balancer is exactly the same. No matter how intelligent your load balancer is, it isn’t clairvoyant. There is always a chance of live traffic being passed onto an node that’s about to become unresponsive, and any traffic routing solution worth its salt knows that and has some kind of transparent retry logic baked in.

Really, the only way the above scenario can be avoided at all times is by broadcasting duplicate traffic to multiple backend servers simultaneously and returning the reply from the winner (à la dnsmasq --all-servers). That’s a lot of overhead that would be better used passing new requests!

Now, this isn’t to say that passive health evaluation is good enough for all use cases. There are real network penalties that get paid, penalties that are minimized in this example since it’s all happening over a local network. And while the worst-case scenario for passive and active health checks is identical, active health checks make that scenario (passing live, non-synthetic traffic to a dead node) less likely. You can also get a lot fancier with active health checks by sending known-good requests and expecting positive HTTP response codes. I don’t particularly see that as the load balancer’s job, but that’s a topic for another time.

This demo proves that open-source Nginx is more than capable enough for my needs. And, more importantly, it serves as reminder to always lab out your assumptions. Chances are you’ll learn something.

WebM is probably the future, but like all things containing the word “web,” there’s an element of nuance and surprise that emerges while working with it.

VP9 is supported pretty much everywhere that matters — even Edge manages to produce a video UI, although one imagines it grumbling as it does so — but the only pixel format that actually works across computers and phones alike is YUV420p. For the most part, this detail manifests itself as an additional flag you must pass to your video encoder of choice (e.g. ffmpeg -pix_fmt yuv420p) lest you present a perfectly playable but nevertheless blank video to some, if not all, of your audience.

But there’s another layer here, one that picks apart the strange name of YUV420p, buried in a drawer somewhere with the label “Chroma Subsampling.” For efficiency reasons, YUV420p will keep only one pixel of color information for every four pixels of video, which means a loss of detail even if you tell it to encode “losslessly.”

For the most part, this produces a perfectly acceptable result, unless you’re doing something that lends itself to pixel precision like screencasting a small terminal window (and, let’s be honest, cultivating a habit of getting hung up on details). The correct solution to this is to move to YUV444p, which keeps a good old-fashioned one-to-one ratio of color data to pixels. Except, like so many “standards,” it produces inconsistent results across browsers. In my limited testing, Chrome and Firefox on Android were able to play YUV444p, along with Firefox on Linux, but Chrome, Firefox, and Edge on Windows didn’t have a clue what was going on and put up a black box instead. Even my file browser wasn’t happy with it, if I’m honest, as it generated a sickly-green thumbnail for the video file.

Options for platform-agnostic pixel-precision are slim, it appears, at least if you’re keeping to strictly HTML5 video. I’ve found a simple way around this for now, though. Since chroma subsampling wants to reduce every 2x2 block of pixels to a single chromatic data point, why not feed it 4x the amount of pixels? And to prevent the filesize from growing unnecessarily, don’t let ffmpeg attempt to do any kind of interpolation when scaling up — stick with the nearest neighbor algorithm and make plain ol’ 2x2 chunks.

This is the best I’ve come up with for now, but I’m hoping YUV444p support will be more of a thing in the future to avoid this kind of workaround.