State of the Nerd Addendum: Failing Fast8 min read

In my previous State of the Nerd update, I talked a bit about how I was investing time and resources into experimenting with Docker, Rancher, and RancherOS. There have been a lot of developments and without going into too much detail, I think it’s worth dropping an update.

Intel NUC6i3SYK

The new NUC arrived. These Skylake NUCs are still a little hard to get a hold of, so to help you out I’m including a breakdown of everything with sources. While I usually buy stuff from Amazon (as a Prime member, I like the quick shipping), Amazon didn’t really have my back on this one so I went with Newegg.

Total with shipping came in at almost exactly $600 at the time of my order, though I paid a little extra on shipping and rush packaging to make sure I had it all in time for hacking over a three day weekend.

freaking love this hardware. I just want to say that. The Intel team did a fantastic job engineering the NUC, and I think Apple should take note. This is a better Mac Mini than the Mac Mini, and at a lower cost. There’s just one problem: it’s not a Mac Mini.

No benchmarks have been run, but the seat-of-the-pants-ometer says this thing is a wickedly quick inexpensive little box. The Samsung SSD really ripped during the operations that were storage-intensive. I can only imagine what this would have been like with an M.2 NVMe SSD (which will work with the hardware, but many operating systems are still playing catch-up to this great new boot disk option).

Curious observation: on the Celeron NUC, the power button lights up. On the i3 NUC, it does not, but a blue light on the front of the chassis does. Otherwise these fifth and sixth generation NUCs enjoy unmistakeable common lineage and look great next to each other or stacked on top of each other.

Rancher & RancherOS

I want to love this stack. I want to love it so much. On paper, they’ve got all of the right parts. It’s so simple to install. The fundamental design has an elegance that appeals to my autistic pattern-thinking brain. But it has fallen short in some areas. Since the stack is still very young, and still considered beta, I’m hoping to return to it when it’s a bit more robust. For now, I’m moving on.

Things move fast & furious in the homelab, so I didn’t take extensive notes. This is a great place in which to try out quick hacks, to explore, to validate hypotheses before diving deeper in a more planned-out way. I will say that I ran into a number of situations with this stack where things flaked out in a way that brought several of my guest services down hard. I felt frustrated trying to figure out in this new paradigm where I can look to debug the problems. Both of my NUCs were reformatted, re-installed, and built up again just in case previous experiments might have been leaving cruft behind that were tainting current ones. At one point, the new NUC either rebooted or its Docker subsystem restarted. I’m not really sure which. Since even the system console runs in a container, I’m not sure what uptime number to believe.

When things got bad, the rancher-agent container would lose contact with rancher-server. It’d get left in a hung state where the two were no longer talking to each other, rancher-agent would be up and running but not communicating to its master. I’d have to manually restart that container to get them talking again. Except at this point convoy-nfs was in a broken state that I didn’t know how to begin to diagnose. Restarting containers didn’t get it back into a good state. Rebooting the machine wouldn’t, either. Only the containers that didn’t have convoy-backed volumes would come up. Those with convoy-backed volumes would just go into an endless start/stop loop. At the console, I could see the NFS filesystem mounted, and I could write to it. I’m not really quite sure where to go with diagnosing problems with Rancher and Convoy, but the inherent fragility of it all left me not wanting to invest the time into figuring it out. The documentation, too, was pretty sparse. So I wasn’t filled with confidence that this stack is ready to show off at work as a platform we should be investing in.

I’ve no doubt that this is going to become a great platform. Right now I see it as a great concept that shows great promise. But I don’t yet trust it even with my homelab experiments. I’m going to fail fast here and move on.

What’s Next?

I wanted to try Rancher first frankly because of the web UI. One of the things I really like to do professionally is offer developers great tools they can use to get stuff done without worrying too much about how the infrastructure works under the covers, or having to go hat-in-hand to the sysadmins and ask nicely pretty please for something (often to have a grumpy sysadmin say “no”). Self-service is important. And while having great command line tools and an API is important, I think it’s also vital for these tools to have a fully formed web UI for rapid prototyping and dashboards.

This is why I didn’t start right out with CoreOS and Kubernetes. But as these are more mature contenders in the container orchestration space, that’s what’s next.

I’d also like to get one more NUC6i3SYK setup into the homelab for the Docker work.

Observations on recent hardware upgrades

One other note: I’d moved from rack mount servers to NUCs in large part to cut down the noise and heat in my home office space. This is working out really well. The Celeron NUC produces no heat that I can detect with the palm of my hand held an inch or two behind its rear vent. The i3 produces only gentle warmth under load. If I place my hand on the i3’s lid, I can feel the vibration of its fan running, but I cannot hear it. My NUC stack takes up about as much desk space as a spool of DVD-R media and makes about as much sound. In short, I love it.

The only fan noise that I can detect now is from my Force10 s50 switch. The HP Proliant N40L might also be making sound, but I can’t hear it above the Force10 switch. The Proliant has a big slow case fan, so I sincerely doubt it’s making much of a contribution. That machine was such a great value when it was new. I wish they were still making it. The current generation comes in at a much higher cost because of the prioritization given to upmarket CPU’s from the cheap but effective AMD Duron.

The Netgate RCC-VE 2440 is working out fantastically running pfSense as my new firewall. I’ve been keeping an eye on CPU/memory/storage/temperature, and all figures have been amazingly healthy. As a fanless machine, it’s dead silent. It’s just important to have open airspace around its vent slots, so I’ve had to avoid the temptation to stack NUCs on top of it. I’ve gradually been adding more and more functionality to my firewall. More recently, I’ve been traffic shaping to prioritize TCP ACK’s, to make sure that video games and media streaming works flawlessly, and to de-prioritize non-interactive traffic like my Bitcoin server. The CPU on my firewall remains mostly idle, with utilization peaks remaining well under 10%. This is on a 100/10Mbps residential broadband connection.

NUC Wishes

If anyone from the NUC team at Intel is reading this, I’d really love it if you could find the room for either dual M.2 NVMe devices in the low profile models, or dual 2.5″ SSD’s in the high profile models. Such a change would greatly expand the roles that this device could play, reaching into home servers and small business servers (RAID!) I don’t think you need to provide hardware RAID functionality. I’d rather not have it. I’d be willing to give up the SD card slot and the headphone jack.

As for the modular lid design, how about using it for an optional chassis extension? With a PCIe SAS controller, connecting to a chassis extender lid, and within that two to four 2.5″ drive bays or (if it’s even possible) M.2 SATA slots. I’d love to use NUCs as little servers but I really need to be able to set up redundant storage devices to do that.

And I want to reiterate: I think the next Mac Mini should be the result of a partnership between Apple and the Intel NUC team. Intel is doing it better, guys. Make some styling changes, add a Thunderbolt & USB C to it, slap an Apple logo on top and OS X inside. I’ll buy it.