APUG database crashed, had to restore from backup..

Discussion in 'APUG System Announcements' started by Sean, Apr 23, 2012.

  1. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Hi All,

    Just when things were starting to ease back into place the most obscure of things crashed the entire database. In my efforts to find and purge old code that was possibly giving us issues, I removed an old plugin. Nothing unusual and something I have probably done 300 times over the last decade. Often if removal of such a plugin code gives grief it is easy to recover. In this case it crashed a key area of the database used to iron out any glitches, something I had never seen before. I had a vbulletin tech on hand a couple hours later who had also never seen such a thing happen. Anyway, after failing to recover the database I fell back to my backups from around 3hrs before. So, you may have some loss of data. As far as I can tell the restore has been successful.

    Times of data loss April 23rd:
    Database Backup occurred at roughly 7pm UTC
    Database crash occurred at roughly 10:25pm UTC
    Data between 7pm-10:25pm will be missing
    here is a time zone calculator if needed: http://www.timeanddate.com/worldclock/converter.html

    The good news is I've made contact with a highly skilled vBulletin tech who is available for some work. After catching up on some sleep I will be turning over some of the big issues to him.

    I am sorry for the recent issues with APUG stability. I will continue to do my best to ensure this is sorted out as soon as humanly possible. Thanks again for your patience.

    Sean
     
  2. andrew.roos

    andrew.roos Subscriber

    Messages:
    568
    Joined:
    Jan 4, 2011
    Location:
    Durban, Sout
    Shooter:
    35mm
    Congratulations on having an effective disaster recivery plan in place. It could have been so much worse.
     
  3. k_jupiter

    k_jupiter Member

    Messages:
    2,577
    Joined:
    Feb 3, 2004
    Location:
    san jose, ca
    Shooter:
    Multi Format
    According to your time zone calculator link, your database mishap won't occur for three and a half hours. Plenty of time to get it fixed.

    I feel like i am in an episode of Dr. Who.

    tim in san jose
     
  4. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    I don't want to get my hopes up yet, but it almost seems the database dump file and import of the backup purged some corruption. As of now the site is performing normally. Before the database restore it took me 20 seconds to get into my PM box (now it's less than a second), it also took gallery images 5-7 seconds or so to load (now within 2). Also our response times are averaging 102ms which has never been this low before (prior to the crash 500ms). If anything it will be interesting to see how long this performance lasts. Will keep an eye on it, but for now.. sleeeeeeep ZzzzzZzzzz.....
     
  5. MattKing

    MattKing Subscriber

    Messages:
    15,193
    Joined:
    Apr 24, 2005
    Location:
    BC, Canada
    Shooter:
    Multi Format
    I blame the deleted thread - or Aristophanes :wink:
     
  6. FiatluX

    FiatluX Member

    Messages:
    223
    Joined:
    May 11, 2010
    Location:
    Scandinavia
    Shooter:
    4x5 Format
    I´ve seen several vbulletin forums crash or being hacked lately, seems that one of the ways is a java enabled iframe! Check any suspicious code as the other forums have had unwelcome exploiters that inserted malicious code which spread viruses to the visitor computers
     
  7. TheFlyingCamera

    TheFlyingCamera Membership Council

    Messages:
    9,180
    Joined:
    May 24, 2005
    Location:
    Washington D
    Shooter:
    Large Format
  8. hrst

    hrst Member

    Messages:
    1,300
    Joined:
    May 10, 2007
    Location:
    Finland
    Shooter:
    Multi Format
    Same with http://www.apug.org/forums/forum205/31480-real-formula-10.html from
    http://www.apug.org/forums/forum205/ ; the top forum shows "tars" has made the newest post as of today, but Page 11 redirects to Page 10 and this new post cannot be seen.

    And wow, thanks for the quick response to the problem, once again. It seems you really care about this site.
     
  9. MattKing

    MattKing Subscriber

    Messages:
    15,193
    Joined:
    Apr 24, 2005
    Location:
    BC, Canada
    Shooter:
    Multi Format
    It must have something to do with how user preferences are set. I have increased the number of posts per page (to 40), and I don't have any problems with all three of the example threads.

    I have in the past, however, had similar problems with other threads.
     
  10. Marco B

    Marco B Member

    Messages:
    2,981
    Joined:
    Jan 13, 2005
    Location:
    The Netherla
    Shooter:
    Multi Format
    Hi Sean,

    I know I don't just speak for myself here, and you've heard it before, but in keeping accordance with one of the above post's SF "Dr. Who" reference: I think your dedication to support the forum considering it's size and complexity, is more akin to "super-human" :wink:...

    Good to hear the backups did their job. I once worked for a major government organization running a huge spatial database with unfortunately outdated and even end-of-life soft- and hardware (one of the reasons I was working there: to provide insight in and plan for a major change and complete replacement of the systems with more modern equivalents). During some routine work, some person noticed the backup hadn't run properly for months on end... Fortunately, the main system had...

    Marco
     
  11. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    That's interesting, I might need to adjust the default for that. Good spotting
     
  12. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Ok am back on line now. Thanks for your support and reporting on a few glitches. The server has been running very good after the database re-import. Response times still under 200ms. I think the database backup and following restore must have purged some deep corruption that the standard database tools could not get rid of. Here's hoping we have some stability for a while and can simply improve upon that. Looks like the crash, although stressful may have helped in the end. We'll see how it goes. There are still several issues with bad database queries (this is from poorly designed code or add-ons). I have a guru coder available to make new, more efficient codes that put far less strain on the server. I will setup a clone of APUG on a hidden domain for any further testing.

    Thanks
     
  13. polyglot

    polyglot Member

    Messages:
    3,469
    Joined:
    Jun 12, 2009
    Location:
    South Austra
    Shooter:
    Medium Format
    One more bit of corruption: clicking the "My Posts" link is showing up a couple of threads ("Identify this old Gitzo" and your friend's masters research thread) that I've never posted in.
     
  14. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    i probably need to rebuild the search since we restored will do that in a few.
     
  15. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    For those having trouble getting to the last page of some threads, I found the fix for that. A vBulletin utility is running right now that updates the thread/page counts. This will probably need to run for another 10mins before it completes, then the last pages should work fine. If not let me know. Thanks

    *Actually just noticed, still some weirdness there with not showing additional pages in search results, investigating
     
  16. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    search indexes have been rebuilt, any weirdness with "my posts" "my threads" should be ok now. At the moment I cut results back to 100 max, but will increase this again soon.
     
  17. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Ok I think all searches are working properly now (my posts) etc
     
  18. eddie

    eddie Subscriber

    Messages:
    1,786
    Joined:
    Jul 24, 2005
    Location:
    Northern Vir
    Shooter:
    Multi Format
    Sean- Thanks for all you do.
     
  19. ozphoto

    ozphoto Subscriber

    Messages:
    1,270
    Joined:
    Apr 2, 2006
    Location:
    Bangkok, Tha
    Shooter:
    Multi Format
    Hi Sean, not sure if you've discussed previously, but I can't customise the sidebar any further than having "Recent Classifieds" appear, even though I've selected "My Recent Posts" to appear.
     
  20. Sean

    Sean Admin Staff Member

    Messages:
    8,914
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Yeah some of the sidebar has been temporarily disabled. Several of the blocks were pegging the database with ridiculous amounts of queries. I have a coder tasked to redo a few blocks so they use far less resource.
     
  21. Ross Chambers

    Ross Chambers Member

    Messages:
    703
    Joined:
    Apr 26, 2004
    Location:
    Blue Mountai
    Shooter:
    Multi Format
    Dear Sean, please come over and work for my bank. When their online facility goes under they don't explain, don't apologise, don't seem to worry unduly at all; just ride it out.

    Many thanks for your hard work.
     
  22. FiatluX

    FiatluX Member

    Messages:
    223
    Joined:
    May 11, 2010
    Location:
    Scandinavia
    Shooter:
    4x5 Format
    Awesome work, thanks!