long weekend in the homelab5 min read

I’d been neglecting opportunities to work in my homelab for awhile so that I might take advantage of a bit of a creative streak and shoot some photography. That’s probably been satisfied, for a little while anyway, so I’ve spent much of this long weekend tinkering in the homelab.

A few things I’ve learned along the way include:

  • tmux is pretty awesome. I’m using it now instead of screen. I also switched from Terminal.app to iTerm2 over the weekend and I’m digging it so far. Though the much vaunted tmux integration isn’t there yet.
  • My respect for Ansible continues to grow. I was able to pretty quickly hack together a playbook to help me get several new virtual machines up to speed.
  • My respect for The Foreman wanes. It seems particularly fragile. The error messages aren’t as clear as they could be, so troubleshooting issues is not as intuitive as it could or should be. Things started going downhill when I installed the foreman_discovery gem. I’ve since tossed Foreman out of my homelab infrastructure. I can’t help but wonder if there is a quick and dirty mentality in the Ruby on Rails community, or if the fragility is inherent to the framework itself.
  • Similarly, Docker isn’t quite there yet. I got bit by #4036 and #5684. I’ll continue to tinker with Docker but I immediately moved the workload from Docker into Virtual Machines.
  • To that end, I stood up Jenkins in a VM along with one build slave. This is a tool I’d like to learn more deeply for unit testing of operations tooling. Just to get the hang of things, I picked a simple to build project (Tor) and set up a simple job to watch the git repo and compile it from source. I’ll start building out a delivery pipeline to test builds on multiple platforms, run unit tests, etc. for the academic exercise.
  • I’ve been throwing virtual machines at my HP Proliant DL160 G6 (12 cores from 2x Intel Xeon X5650‘s, 72GB RAM, 2TB guest storage). Load average stays well below 2.0 most of the time. I like this box.
  • I have another server, a Dell PowerEdge 1950 III,  in the cabinet right now that’s a “utility class” machine. No virtual machines. Right now it’s hosting DNS, and I expect to add dhcp once I can deprecate the Apple Airport Extreme from routing duties. There’s some wiring I have to do in the house to facilitate this, and I have to rethink my wireless & VLAN strategy a bit. This box is nowhere near as decked out but it should be more than enough to handle basic services. I’m thinking I might jack up the RAM, though, and move the basic functions off into virtual machines there. The original plan was to use Docker containers, but I don’t think the technology is quite mature enough yet.
  • I’ve prototyped a Tor-only VLAN with the idea that I’m going to open it up for WiFi access. Hosts on this VLAN can only exit the subnet through a transparent Tor proxy. With the nearly complete loss of privacy in the United States, I think it’s important for us engineers to spend some of our time extricating the world from the horrible quandary we’ve had a part in creating.

What are some of the things coming soon?

  • Monitoring. I hate monitoring. It’s a necessary evil, but my peers have held on to awful tools for far too long (see #monitoringsucks). I’m looking squarely at Nagios and check_mk here as prime examples.  I’m going to take Sensu for a spin.
  • Measurement. Measure all the things. Will almost certainly use graphite and grafana. Going back to what I said about awful tools, I definitely won’t be using Cacti. Why won’t these old school tools just roll over and die already?
  • Log aggregation. Dumping everything to one syslog server is easy but that’s not enough. Lots of shops use Splunk but I’m going to use an Open Source solution. Probably some combination of logstash, elasticsearch, and kibana.
  • Authorization/Authentication. I still have to get LDAP and Kerberos up and running.
  • Beefing up Jenkins. A big part of why I have a homelab is to learn more deeply the technologies that I barely get to touch at work. This ends up helping me to make better decisions at work. I’d really like to take the time to make fuller use of Jenkins and get it dancing a jig for me.

And further out?

  • Storage. I’d like to get some kind of redundant storage service going, providing both a block store and an object store. This is going to require some hardware upgrades, first of all. Software-wise, I’d like to try ceph. I do have a somewhat more immediate need to get something like gluster going for a shared filesystem.
  • Home Entertainment. I’ve bought hundreds of DVD’s over the years, maybe over a thousand. I’ve bought oh so many music CD’s. And I’ve got access to both a great array of shows on cable TV and over-the-air broadcast. Using the AppleTV for large media libraries is a really clumsy experience. Also, it’s not at all useful for accessing shows in a DVR. Once I have storage worked out, I might like to see if I’m in a better position to replace the AppleTV and the DVR’s around the house with something more appropriate.
  • UPS. Right now these servers are plugged into a surge protector (eek, yeah, I know). I need to invest in a proper UPS for them.
  • Switch. My desktop and server cabinet are sharing a Cisco SG300-10 switch. I’d like to get a more serious switch into the cabinet and leave the SG300-10 just for my random collection of desktop/deskside systems.
  • Another hypervisor. Once storage is worked out, I’d like to get another one of these DL160 G6’s so VM’s can be live migrated and I can do maintenance on hardware without taking things down.
  • Backups. Right now it’s all so manual and clunky and not very DevOpsy. (The cobbler’s children have no shoes.) I’m going to set up some regularly automated off-site backups and a rotation. But I need to throw a little bit of hardware at this.