APUG.ORG OUTAGE NOTICE

[roteague]

Thanks Sean for all your hard work and dedication.

[MikeM1977]

Although failures happen to everybody, this is why there is still a very large market for expensive machines from Sun, IBM, and other providers who do more than assemble x86 boxes.

[JeffD]

Good time to review the backup procedure. I can't figure out how a failure on the server would screw up your backup, but I guess it did. I started backing up my web server off site daily, after an event similar to yours convinced me it was essential. Now I sleep better at night.

[Sean]

Good time to review the backup procedure. I can't figure out how a failure on the server would screw up your backup, but I guess it did. I started backing up my web server off site daily, after an event similar to yours convinced me it was essential. Now I sleep better at night.

Well I had 4 backups in play luckily and between them we were able to recover everything.

1 backup on server's secondary drive (it was partially corrupted as it was a copy of the corrupt primary disk)
1 decommissioned server in california (it's data was good but from the 15th)
1 off site full rsync backup (data was good but only had apug.org and was from the 17th)
1 full site backup from the 19th on my local pc with good data but hard to transfer with my slow NZDSL connection

The problem is most of the data was corrupt in production, and that corrupt data then copied over to the nightly backups which made the backups corrupt. We had no indication there was an issue until the crashes. When you have about 6.5gigs of data in the backup you never truly know it's good until you try to use it. I never anticipated this because usually a primary disk dies and you fall back to the secondary with last nights data. In this case last nights data was also bad. Luckily we had a variety of backups + acquiring some still good db's off the old disks.

In the future I plan to not overwrite nightly backups anymore, for example maybe have 15days worth of backups on the backup disk. If data goes bad we can keep going back a day until it's clean (hopefully always having a recoverable set of databases less than a day old). I'll be examining a lot of options. At this very moment I have a newly generated zip of the entire site transferring to my offsite storage and my local pc. I'll then take nightly database backups until the new solution is in place..

[Sean]

-also working on a failover page so in the event of an outage we can have a notice with temporary chatroom.

[Markok765]

-also working on a failover page so in the event of an outage we can have a notice with temporary chatroom.

Would the chatroom be open only to subscribers? Thanks for the hard work!

[mikebarger]

Thanks Sean, I thought maybe I was on everyone's ignore list

Mike

[MikeM1977]

Well I had 4 backups in play luckily and between them we were able to recover everything.

What is your database? MySQL?

[Aggie]

Dell Sucks! (I like the word sucks it is so to the point) I too have had one of their laptops, which was delivered without all the parts ordered on it. Everytime I had to try and get the proper part they would tell me "You get what you pay for" to which I would reply, "No I didn't that is the problem." It took 8 months for them to send me the correct parts, and then it became a doorstop due to multiple failures withint another 8 months. My son had a desktop from them. He had more problems than I did. He finally gave up and built one of his won from scratch.

[Sean]

What is your database? MySQL?

yup