Sometimes When Things Go Right, They Still Go Wrong

by the other theo

One of the trials of having a small child is the button pushing.  Thankfully, the Peanut is not too much of a trial in this regard.   He pushes buttons occasionally, but not enough to be a serious problem.

We have a dedicated file server in the house.  It sits under my desk in the “office”.  It’s only got one button on it, the power/reset button.   The Peanut hit it last week.   That normally shouldn’t be much of a problem.   In this case, powering the system on caused one of the fans to sound like a small kitchen appliance.   Worse yet, it was one in the power supply.

So, I took an hour or two last weekend and replaced the power supply.  Now I can barely hear the system, but there was another problem.

I decided to put a monitor on the computer to verify that everything started correctly after the new hardware was installed. When I did, I discovered that there were some errors worth bothering about:
godzilla smartd errors

This was not good.  It was one of the disks that hold my iTunes library and the collection of SVG files that the Missus uses for crafting.

The good news that the disk is part of a two disk mirror using the ZFS file system on FreeBSD.   While the disk itself was reporting errors, the ZFS file system reported no errors whatsoever.

The disk in question was only(!?!) 1TB and getting a little on the full side anyway.

So, I bought two 2TB disks and replaced the defective disk and then the other disk in the mirror in two separate steps.   ZFS faithfully copied the data from the existing disk in the mirrored pair each time.  Easy.

Well maybe not so easy because I had this problem when I was done:

zpool data status

This happens because newer disks organize data in 4KB sectors, not the traditional 512B.

Fixing it required me to do what I’d hoped to avoid: backing up the data, wiping out the data zpool, and restoring it from backup.

Thankfully, ZFS doesn’t make doing that terribly difficult.   Thanks to eSATA it only turned out to be a morning’s worth of work.