This is just a quick check-in round to cover an assortment of subjects.
Last time I checked in, I let you in on my dirty little secret about loving cheap fountain pens. Well, that was so much fun that I went ahead and set up a new blog just to focus on great values in analog writing tools (especially fountain pens, inks, and paper). Enter: Chronicles of a Cheap Geezer.
I’ve been pretty quiet about the homelab. There’s a good reason for that: all that rackmount gear that I invested in ran very hot and jacked up my power bill. So I mostly left it shut off. The heat in my small home office had become quite unbearable. What have I been doing to remedy the situation?
- I had been running pfSense on a Dell Poweredge 1950 III. That’s a pretty old server, and way overpowered for my 100Mbps downlink. It wasn’t worth the space, the heat, or the consumed electricity, let alone the footprint. I replaced it with a Netgate RCC-VE 2440.
- The HP Proliant DL160 G6 12-core monster was the worst offender. That was always meant to be one of several servers for standing up a private cloud. The plan ended quickly when just the one server alone drowned my workspace with heat. It’s a real beast, but I’ll have to go with something more efficient. Enter…
Virtualization is so 2006. The industry has moved on. If you’re not investing a lot of your internal R&D into containers by now, you’ve got a lot of catching up to do.
The big player in the container field is, of course, Docker. But how to best implement Docker in my homelab without firing up the DL160?
I’m reswizzling my homelab around Intel’s NUC line of products. The first machine to join the homelab was a humble NUC5CPYH, into which I added 8GB of RAM and tossed in a 120GB SSD that I had laying around. This is where I started playing with Rancher, which I’ll get to shortly.
The NUC5CPYH is a really low power system, and now that I’m moving everything to Docker and Rancher, this system is going to effectively be the management node for everything else. Aside from Rancher itself, I’ll likely only run a few other lightweight containers on it. Using node labels, I can force some containers to run on this host, and most others to run on other hosts.
Those other hosts are the slightly more powerful NUC6i3SYK, into which I’m adding 32GB of RAM and a 250GB SSD. While these machines support M2 NVMe storage, SATA SSD runs at half the cost and is pretty well-supported. Intel makes an i5 version of this machine, but it sips a little more power and adds $100 to the cost of every node. I expect to run two of these nodes in the homelab.
If that doesn’t sound like much storage, that’s because the stateless containers won’t use much local storage, and on any host that will require persistent volumes, I’m using my trusty old HP Proliant N40L microserver running OmniOS to provide a backing store on ZFS via Convoy NFS.
I’d already mentioned Rancher. I’m using Rancher to manage a fleet of Docker nodes. Not only does it have a nice web interface, but it also has a rich API. Using configuration files to define entire application stacks and then deploy them is pretty straightforward with its tooling.
Rancher can run on any Linux distribution that supports recent stable releases of Docker, including one of their own purpose-built design: RancherOS. The distribution is only 20MB. I installed it from a USB thumb drive in minutes. It boots in seconds. You get a kernel and then instead of starting systemd, it goes straight into Docker! Every system service runs in a container, including another instance of Docker! So you’ve got your system Docker doing all of the stuff that an OS would normally do, and then you have another Docker running within it where you put all of your own containers.
So the first container I installed was the Rancher manager, which had me up and running in a couple of minutes. After opening a port on my firewall to allow outside nodes to tickle Rancher manager’s API, I was able to start spinning up cloud instances on DigitalOcean and bootstrapping Rancher onto them. Entering not much more than my API key and ssh public key, I spun up 9 droplets that all showed up in my Rancher console within about half an hour.
To be honest, for homelab purposes, running much of my persistent containers in a public cloud will become quite expensive. There’s a big up-front cost to buying my own hardware, but I can do a lot more with it. I get especially screwed on storage in the public cloud. I expect that might change someday, but for now the hybrid cloud approach that I talked about four years ago still makes a lot of sense.
I like RancherOS. It’s still very beta. The documentation is still very sparse. Having an OS that runs everything in Docker will take some getting used to. When you log into the console, you’re poking around inside of a container. When you ssh in, yup, you’re in a container. So some of the traditional SysAdmin troubleshooting and management skills don’t apply here in a straightforward way.
I’m running into some annoyances right now with long standing bugs in Linux’s IPv6 stack vs Duplicate Address Detection. This is the only real problem in RancherOS that I haven’t yet taken the time to work out.