APUG.ORG OUTAGE NOTICE

Discussion in 'APUG System Announcements' started by Sean, Jul 22, 2006.

  1. Sean

    Sean Admin Staff Member

    Messages:
    8,967
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    APUG.ORG OUTAGE NOTICE

    We are recovering from catastrophic hardware failure which had also
    unfortunately corrupted our primary data and backup data (worst case scenario pretty much). We have been working around the clock to get the site back up. Currently we have recovered a good filesystem from the 15th and good databases from just before the crash. The problems began with faulty hardware, surprisingly on a brand new high end server. Possibly a combination of faulty ram and a faulty primary disk. Over the past day or so the owner of the hosting company has personally worked with me and other
    unix techs on the issue for over 30hrs straight. We are now on a new
    server and partially up and running. The host is very distraught over this outage and is sending the server back to Dell demanding a full analysis and 'post mortem' as they call it in IT. He has also given us a month free of hosting. This is some horrible bad luck we've had and luckily our provider did not just give us a "sorry, your data is corrupted nothing we can do" and leave us hanging. They spent about 12hrs working on a disk to recover our databases not to mention setting up a new server as quickly as possible. Needless to say many new things will be put in place to ensure a better disaster recovery system for us. Still working on things so better run. Thank you for your patience, very sorry for the length of the outage.

    Sean
     
    Last edited by a moderator: Jul 23, 2006
  2. MurrayMinchin

    MurrayMinchin Membership Council

    Messages:
    4,196
    Joined:
    Jan 9, 2005
    Location:
    North Coast,
    Shooter:
    4x5 Format
    Quit being a Hero and go have a nap :smile:

    Murray
     
  3. JLP

    JLP Subscriber

    Messages:
    1,610
    Joined:
    Feb 22, 2006
    Location:
    Oregon
    Shooter:
    Multi Format
    Perfectly said Murray.
    Keep up the good work Sean but don't burn the candle in both ends.

    jan
     
  4. DBP

    DBP Member

    Messages:
    1,895
    Joined:
    Mar 22, 2006
    Location:
    Alexandria,
    Shooter:
    Multi Format
    Take a breather. At a certain point working becomes less productive than resting.
     
  5. ricksplace

    ricksplace Member

    Messages:
    1,565
    Joined:
    Jan 22, 2006
    Location:
    Thunder Bay,
    Shooter:
    Multi Format
    I was told once the best way to fix a faulty server after a catastrophic failure such as you describe.

    12 guage with 00 buckshot.

    Thanks for being there when Murphy's Law strikes.
     
  6. CRhymer

    CRhymer Subscriber

    Messages:
    442
    Joined:
    Feb 8, 2005
    Location:
    Fort Smith,
    Shooter:
    ULarge Format
    Thank you for all your work. One really appreciates what this site has to offer when it is offline for even a short while. I was starting to get the DT's. Hope you get a rest soon.

    Cheers,
    Clarence
     
  7. copake_ham

    copake_ham Inactive

    Messages:
    4,090
    Joined:
    Jan 26, 2006
    Location:
    NYC or Copak
    Shooter:
    35mm
    Bummer to be sure. Glad site is back up and thanks for working so hard to do so. I missed this place while it was down. :sad:
     
  8. Jim Chinn

    Jim Chinn Member

    Messages:
    2,512
    Joined:
    Sep 22, 2002
    Location:
    Omaha, Nebra
    Shooter:
    Multi Format
    No wonder Dell's stock price took a major dive into the crapper on Friday! :mad:

    Thanks for all your efforts Sean. When the gremlins hit, it always seems to be worst case scenario in my experience.
     
  9. Bill Hahn

    Bill Hahn Member

    Messages:
    351
    Joined:
    Nov 19, 2004
    Location:
    North centra
    Shooter:
    35mm
    Ditto to what CRhymer said.
     
  10. Donald Miller

    Donald Miller Member

    Messages:
    6,242
    Joined:
    Dec 21, 2002
    Shooter:
    Large Format
    Thanks Sean, Your efforts are appreciated. Have some well deserved rest.
     
  11. naturephoto1

    naturephoto1 Subscriber

    Messages:
    1,819
    Joined:
    Mar 26, 2006
    Location:
    Breinigsvill
    Shooter:
    Multi Format
    Sean,

    Sorry for your headache, but as others have said, I missed this place. Hopefully you can get some rest soon and you do not experience anything like this again any time soon.

    Rich
     
  12. Joe Lipka

    Joe Lipka Member

    Messages:
    809
    Joined:
    Sep 24, 2002
    Location:
    Cary, North
    Shooter:
    4x5 Format
    Well, as the IT folks will tell you, the failures either happen right away or much, much later. (But you knew that.) Thanks for your effort in getting life back to "normal" here.
     
  13. Photo Engineer

    Photo Engineer Subscriber

    Messages:
    25,102
    Joined:
    Apr 19, 2005
    Location:
    Rochester, N
    Shooter:
    Multi Format
    I would like to applaud Sean and his networking organization for diligent and dedicated work reconnecting us all!

    Thanks.

    PE
     
  14. Sponsored Ad
  15. bnelson

    bnelson Member

    Messages:
    7
    Joined:
    Jun 26, 2006
    Shooter:
    Medium Format
    I for one missed this site when it was down. Keep up the good work and thanks for working on this hard problem
     
  16. papagene

    papagene Membership Council

    Messages:
    5,282
    Joined:
    Jun 11, 2004
    Location:
    Western Mass
    Shooter:
    Multi Format
    A great big THANK YOU to you Sean. Again you proved the BEST forum operator around.
    Get some rest... spend some good time with your family... and then go out and expose some film!!

    gene
     
  17. David A. Goldfarb

    David A. Goldfarb Moderator Staff Member

    Messages:
    17,922
    Joined:
    Sep 7, 2002
    Location:
    Honolulu, Ha
    Shooter:
    Large Format
    Thanks again, Sean, for getting APUG back up and running!
     
  18. Sean

    Sean Admin Staff Member

    Messages:
    8,967
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Thanks for the comments everyone, hopefully we are in the clear now.
     
  19. papagene

    papagene Membership Council

    Messages:
    5,282
    Joined:
    Jun 11, 2004
    Location:
    Western Mass
    Shooter:
    Multi Format
    cool... even the chatroom is working!! :D

    gene
     
  20. reellis67

    reellis67 Subscriber

    Messages:
    1,887
    Joined:
    Mar 10, 2005
    Location:
    Central Flor
    Shooter:
    4x5 Format
    I'd like to add my thanks for your hard work, this site is an oasis in the desert. Thanks as well for all the work we don't hear about. (I will refrain from giving my opinions of Dell however...)

    - Randy
     
  21. naturephoto1

    naturephoto1 Subscriber

    Messages:
    1,819
    Joined:
    Mar 26, 2006
    Location:
    Breinigsvill
    Shooter:
    Multi Format
    Randy,

    Don't forget, Dell only assembles computers from parts from many different sources. Others make the parts that fail (ed) and they should the accountable party. We will not know what failed until the unit is analyzed.

    Rich
     
  22. reellis67

    reellis67 Subscriber

    Messages:
    1,887
    Joined:
    Mar 10, 2005
    Location:
    Central Flor
    Shooter:
    4x5 Format
    True, but I deal with their problems on a daily basis (117 servers in the data center to date, 23 more on order) so I've seen their idea of quality control. They sell a product with a warranty and a claim of a certain level of performance, and in my experience, they are overly optimistic about that level of performance. Just because they don't make to component doesn't mean they have no responsibility for failures.

    - Randy
     
  23. wilsonneal

    wilsonneal Member

    Messages:
    578
    Joined:
    Feb 14, 2006
    Location:
    Northern NJ
    Shooter:
    8x10 Format
    Wanted to say that while I missed access to the site, the outage wasn't inconvenient. No worries. Thanks for working to get it back up so quickly.
    N
     
  24. Q17

    Q17 Member

    Messages:
    71
    Joined:
    Jul 10, 2006
    Location:
    Minneapolis,
    Shooter:
    Multi Format
    Thanks for all of your hard work, Sean! Excellent job getting it going again! Although I missed having the site here, it prompted me to get out, shoot some photos and go to the Diane Arbus show.

    Now, get some sleep! :smile:

    =michelle=
     
  25. Krockmitaine

    Krockmitaine Subscriber

    Messages:
    505
    Joined:
    Nov 1, 2005
    Location:
    Montréal
    Shooter:
    Medium Format
    Thanks a buddle for all the efforts to put everything back to normal. Must have been quite a stressful day that you had. You've earn the respect of everyone.

    And get some sleep, we'll survive. :smile:

    Marc
     
  26. John Bartley

    John Bartley Member

    Messages:
    1,399
    Joined:
    Sep 12, 2004
    Location:
    13 Critchley
    Shooter:
    8x10 Format
    Absolutely amazing !! The dedication that Sean gives in support of a forum that (for most of us) is a recreation site is above and beyond praise.

    Thank you !!