devices have different sector alignment2 min read

[root@dogface ~]# zpool replace zones /dev/dsk/c0t50014EE2072AB3D4d0 /dev/dsk/c0t50014EE2B66D775Ed0
cannot replace /dev/dsk/c0t50014EE2072AB3D4d0 with /dev/dsk/c0t50014EE2B66D775Ed0: devices have different sector alignment

And that’s where things went wrong.

Oh, it all started as a really great weekend to be in the homelab. Four new Western Digital Red WD60EFRX 6TB disks had arrived, and I was ready to replace my four 2TB disks with them. I’d replaced the first two of four disks without a hitch. They were Western Digital Red WD2002FAEX 2TB disks. I’d been assuming for some time that I had four identical disks in this zpool, but forgot the compromise that I’d made way back when I first set this machine up: I’d used two Red disks and two Black disks (Western Digital Black WD2002FAEX 2TB). So when I tried to replace disk #3 of 4 in my zpool, things didn’t go as smoothly.

devices have different sector alignment

This changes everything. Those WD2002FAEX disks have a 512b sector alignment, while the new WD60EFRX disks have a 4k sector alignment. This fundamental difference in geometry was a tech debt landmine waiting for me to step on it.

I’m in the midst of a shell game right now that’s going to take days. I’m performing a recursive zfs send to a temporary pool that can handle the full contents of the zpool, and will rebuild the pool from that backup. There are ways to get by with no downtime, but this is okay as it’s a home server with nothing more critical than some DVD backups and a minecraft server on it (so far).

As part of this restructuring effort, I’m switching this HP Proliant N40L Microserver from SmartOS to OmniOS. I’ve already replaced the onboard SATA controller with an LSI 9211-8i SAS host bus adapter. The 5.25″ optical disk bay is going to be repurposed with a 4×2.5″ disk cage. 2x Samsung 850 Pro 128GB SSD‘s will be used as the root pool mirror (where the OS boots from). One more of the same type of disk will be used as the ZFS separate intent log (slog) device. I’m saving the fourth bay for a Samsung 850 Pro 1TB SSD (which I haven’t procured yet) to act as the ZFS cache (L2ARC) device.

Why put such an investment into such a silly little potato of a server if it’s just going to host a minecraft server and some DVD backups?

Because its mission scope is expanding. This server will host the backing store for a number of virtual machines that I’ll be running on several much more CPU and Memory equipped hypervisor systems. I’ll be writing more about that later.