Dealing with Corrupt databases

Hmm, the PostgreSQL database was corrupt and I had to manually delete a few records to get everything to the stage where it would recover it's onw integrity. So I lost a few comments. I manually retrieved some of the later ones, from google caches, and the various backups here.

So, what happened? As far as I can tell, the regularly scheduled backups at 0400 on Thursday morning caused the primary drive (old 40Gb) to fail and also completely bricked the primary backup drive (250GB). My Tertiary backup (300Gb) picked up some corruption as well, probably by copying stuff from a failing drive.

Three live copies of my data in on three physical drives and it nearly wasn't enough. However, to be fair, the stuff that was corrupted was in memory or non-comitted stuff in a database. All my client work, designs and reports are safe as far as I know.

I've whacked a new SATA drive in as primary with a fresh install of FC6, and moved services and data over as necessary. A better solution is obviously required: not just copying files, but a proper dump and restore backup, so a failed drives just means swapping connectors around, not two days of trying to remember everything I ever knew about Linux System Administration.