Well, I just went to synchonise my local gallery with my offsite hosting, and tried to search my home folder for the command I usually use. Who tell me do dat?

30 seconds after I pressed Ctrl-C to cancel it, nothing had happened. That’s usually a bad sign, as on an unloaded servr, grep stops quite quickly. So I loaded another SSH session, and tried to get a file listing in my home folder. That session hung too. Things started to ring little alarm bells in my head.

So I logged in a third SSH session, and queried dmesg to see what the kernel had logged recently. This is when the alarm bells turned into full air-raid sirens – ATA and SCSI errors about unrecoverable reads, and failure to reallocate sectors on failed reads. This means the drive that my home directory sits on is dying. S.M.A.R.T. blithely says ‘This drive is fine chief, what you so worried bout?’ At this point, I’ve got a transfer running to copy my 70 GB of images over to another machine with a large enough drive to hold them.

Since this server is using an LVM configuration, and not RAID, I’ve got no redundancy on my disks, nor do I know precisely what data is on what drive. The operating system itself is on a different drive, but it looks like I’ve got close on 120 GB of data on a failing drive, and not enough archive space to back it up. Time to scratch my head, and work out how I can keep this server intact, while upgrading the storage for the images etc to RAID. Though, with a failing drive, that’s going to be fairly hard, as I only have one other spare drive :(

This post written in Lynx.