illumos makes a comeback in the homelab5 min read

Up until a couple of years ago, I was becoming increasingly active in the illumos community. I’d given a talk on the subject at Triangle DevOps, and indeed my most popular entries on this blog tend to be the ones relating to SmartOS. But something happend in my professional career, a conflict of interests, that compelled me to pull back from that community for awhile. The conflict is now gone, and hot on the heels of illumos Day 2014, my interest is re-invigorated.

The homelab was in a bit of a clunky state. Every time things started getting cool, my Apple Airport Extreme would crap its pants and fail me in strange ways. They can supposedly handle up to 50 clients, but I was killing it with fewer than half of that. I haven’t fixed that problem yet, but I know what I’m going to do. More on that later.

Those of you who’ve been following for awhile know that I’ve got a Dell half-rack in the house, and while it is very lightly populated right now, the “big” hypervisor box is an HP Proliant DL160 G6 with 12 cores and 72GB of RAM. It’s been set up in several configurations, all of which left me wanting for something more elegant.

Tonight, I took the machine apart, extracted the HP RAID controller, and replaced it with an LSI SAS HBA. While illumos can handle the RAID controller just fine, ZFS prefers to have a direct view of every disk.

My distribution of choice for this host is OmniOS. SmartOS is also a really neat OS. I’ve had SmartOS running on my HP Proliant N54L microserver for over 2 years now and it’s been rock solid. I have two main reasons why I’m not expanding the SmartOS footprint in my house right now (that could change on a whim):

  1. I use Ansible to manage my home infrastructure. It can manage almost anything it can ssh into, as long as there is a Python interpreter of some reasonable vintage on the other side. The SmartOS non-global zone has no Python interpreter.
  2. I use IPv6 extensively in my home. In fact, IPv4 is the second class citizen here. IPv6 gets more use internally than IPv4. However, in SmartOS, IPv6 is quite a bit trickier to use. Other illumos flavors make IPv6 incredibly easy to set up.

I fired it up with a pair of 146GB 10KRPM SAS disks that I got on eBay. This is for my root ZFS pool (rpool). The root pool is limited to one virtual device (vdev) which effectively means the capacity will never be larger than the smallest disk in the pool, but I can mirror that device if I want to.

For this reason, I’ve also added 4x 1TB near line SAS disks for my zones ZFS pool. I’ve configured this pool in two mirrored pairs.

There are two more drive bays that have been left empty. There will be at least one SSD added to the zones ZFS pool at some point to take advantage of the hierarchical capabilities of ZFS.

# zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 37.9G in 0h43m with 0 errors on Wed Oct  1 02:42:51 2014

        NAME                         STATE     READ WRITE CKSUM
        rpool                        ONLINE       0     0     0
          mirror-0                   ONLINE       0     0     0
            c3t5000C5000EEC9B91d0s0  ONLINE       0     0     0
            c2t5000C50006176C41d0s0  ONLINE       0     0     0

errors: No known data errors

  pool: zones
 state: ONLINE
  scan: none requested

        NAME                       STATE     READ WRITE CKSUM
        zones                      ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            c1t5000C50056E1D717d0  ONLINE       0     0     0
            c1t5000C50057BE4CCFd0  ONLINE       0     0     0
          mirror-1                 ONLINE       0     0     0
            c1t5000C50057C45E7Bd0  ONLINE       0     0     0
            c1t5000C50057C45F7Fd0  ONLINE       0     0     0

errors: No known data errors

The OmniOS installer did not mirror the root pool by default, but there are simple instructions for setting this up. Also, the network interface is not configured by default when you first install OmniOS, but it’s super easy to do that, too. I installed the latest r151012 release, which just came out a few days ago. Of course, there was a new bash package waiting for me. A quick pkg update took care of that.

One of the things I really like about this platform is there is nothing extra to install to make use of containers (also called zones in illumos). I pretty much immediately started going to town, building zones on this new box. The only real speed bump I’m hitting is standing up a Jenkins container. I’ve heard that this can be a little tricky, but it’s getting late, so I’ll likely hack on that tomorrow night. I’m setting up Jenkins and some build slaves in order to start building out an OmniOS IPS package repository for all of the software that I care enough about to build my way. Actually, probably multiple repositories; there are some I’ll want to share publicly, I’m sure.

I kind of feel like I need to blog more about this stuff, because I know a lot of super smart engineers who just, for whatever reason, don’t know that there are more tools out there to be used beyond Linux. Linux is great, it has its areas where I will reach for it first. But there are also places where I’d rather run illumos. Hopefully, as I write about it here, curiosity and illumination may follow in others. We’ve not even really started here yet. I’ve just gotten the base OS installed, the ZFS pools set up, and basic networking set up.