About a week ago, our home network had a problem. One of the drives on our NAS’s RAID failed. For unknown reasons, rebooting the NAS caused a total RAID failure and everything collapsed. Being a good and wholesome, if amateur sysadmin, Louise (@geekylou) acted quickly to get a replacement drive, and when it became obvious the RAID-enclosure was dead, a replacement box. It was back up and running in a couple of days, and she checked and fixed the backup scripts. Everything was good.
Except, of course, that it wasn’t.
A couple of days later our Linode fell over. This was completely separate, the disc was full up and logs or something broke it. Anyway, we logged in, and we fixed it. Everything was good.
Except, of course, that it wasn’t.
I am not sure if it was Friday or Saturday, but Louise noticed a problem in the backup and repaired it. Something to do with user IDs I think. In any case, she corrected it and did a test restore on a file or two to make sure it worked. All was great.
Except, of course, that it wasn’t.
Today, the NAS fell over. Louise looked. She was already in the computer room scanning some documents. All three discs in the RAID had been summarily kicked for being in a failed state. You can’t make this stuff up. Anyway, she tried to correct the issue and the whole thing fell over in a dirty, smelly, heap of spinning rust.
Every attempt to rebuild the RAID failed, and, since the day was getting on and we were getting hungry, we made the decision to nuke the RAID, reformat and restore from the (previously tested and good) backups. Louise did so, being the designated sysadmin of the household, and began the process.
It is a completely unknown factor that caused Louise to be the Sysadmin. She’s never done the job in real life, being mostly an embedded programmer, whereas I spent most of my working life as a Sysadmin or Helldesk Minion. Maybe it was a case of “Never Again, even though Hell itself reports a problem”, but that is how it is, or was. Nevertheless, backups are now restoring.
Except, of course, that they aren’t.
For some reason. I have asked but… /home was not properly backed up. That is to say, the *symlink* to /home was backed up, but not properly and the home folders weren’t backed up and therefore, not restored. Louise has lost a lot of data. I have lost everything from before she took over the system.
I used to do my own backups. When we installed the NAS and Louise took over the admin thereof, I moved everything from my old archive to the NAS, and then reformatted my old archive disc, to use as a long-term storage for my desktop PC. A 4TB disk of spinning rust to supplement my SSDs. Everything from the time before I met Louise, is gone. Most of it, I probably never want to see again, but there were other things on that drive in /home on the NAS. I mean, it was the most secure storage I had in the end… So my password safe lived there, as well as the master file for my 2FA.
These are all gone, unless a backup, however old, can be found somewhere… anywhere. I have some stuff on Google Drive, and some (though less) on MS One Drive. But most of my digital life is lost and will need a rebuild.
I had protections in place against viruses, ransomware, and everything. All to be lost to an easily fixed backup script failure.
For now, I live in hope.
To be continued…
