Bit of grief with site stability today..

Discussion in 'APUG System Announcements' started by Sean, Feb 21, 2005.

  1. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Hi guys, for some reason the application blocking all of the search and spam bots quit working (leaving us vulnerable to bot attacks again). I have raised a call with that company for help and am just waiting to hear back. In the meantime it is a game of cat and mouse to manually ban the bots giving us trouble until I can get the app back up. I may have caught the few giving us issues today, if not we'll have to go offline again. Thanks, Sean
     
  2. Kevin Caulfield

    Kevin Caulfield Subscriber

    Messages:
    3,406
    Joined:
    Aug 3, 2004
    Location:
    Melbourne, A
    Shooter:
    Multi Format
    Thanks, Sean, for getting it up again, and also for providing the temporary chat room for we desperadoes.
     
  3. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    *update*

    I haven't gotten the bot blocking application back on yet, but I did do a few things that seem to have us stable (cleaned up some code, removed old unused tables from our database, and ran a full repair against all tables in the database). So far it's been over an hour without an issue. I have re-opened the gallery as well. Thanks
     
  4. Michael Mutmansky

    Michael Mutmansky Member

    Messages:
    345
    Joined:
    Sep 7, 2002
    Location:
    Sacramento
    Shooter:
    ULarge Format
    Sean,

    Thanks for the effort...

    I was wondering, would it be possible to eliminate the bots by making everyone go through a hompeage where the access is granted through an encoded password like those used when people are registering an ID on some places (you know, the code is a graphic with fuzzy lines and other junk so a computer can't automatically read the file)?

    Then, you change the access on a semi-random pattern, with semi-random codes. Once you enter the code, it updates your cookie, giving you access to the entirety of the site. The next time you try to enter, the code won't be up to date, and you'll have to manually enter a new code. If the code changes while someone is browsing, it'll pop up the access page when they try to load a new page after the code change. Manually enter the new code, and you're on your way.

    I'm not a programmer, so tell me how stupid of an idea that is.


    ---Michael
     
  5. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Well, that would probably work, but it would then block the site to all visitors and guests. It would be quite easy to totally lock the forum and gallery so that only registered users can access them, the big drawback though is denying visitor access. My long term plan is to get all bots under control, then open us to a few specific bots every now and then (yahoo, google) just to keep our indexes on the net. The net is turning into a warzone though, so I would not be surprised if many sites start opting for a registration only access model to keep all the bots and spammers off their sites..
     
  6. Baxter Bradford

    Baxter Bradford Member

    Messages:
    772
    Joined:
    Nov 17, 2004
    Location:
    Lymington, S
    Shooter:
    4x5 Format
    Are you sure that the threat was purely external?

    It might be that the site itself was expressing its distress about all of the opinionary fisticuffs and fireworks that were happening over in "Ethics and Philosophy". I'm surprised Sean that you weren't contacted by Don King to try to get that thread (where academic brilliance threatened to overtake the thread subject matter) as pay per view..........
     
  7. Digidurst

    Digidurst Subscriber

    Messages:
    629
    Joined:
    Feb 16, 2005
    Location:
    SC
    Shooter:
    Multi Format
    Hi Sean :smile: You probably already know this but just in case, I'm not getting anything when I click on the links for either 'Chemistry Recipes' or 'Articles'.

    Thanks for working so hard to get the site back up! Take care and cheers!
     
  8. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Thanks for the heads up, it's fixed now
     
  9. Ed Sukach

    Ed Sukach Member

    Messages:
    4,518
    Joined:
    Nov 27, 2002
    Location:
    Ipswich, Mas
    Shooter:
    Medium Format
    Ah, yes. There is a LOT of "Making waves" over there. Consider, though; the waves are what keeps the water clear. Without them --- well ... think of a swamp.

    I had just completed a brilliant, and involved, post when .... "Pardon the Inconvenience While ...", came up on the screen. Not a "bad" thing though - gave me - FORCED me - to reflect on what I was writing, and being concise.
     
  10. Will S

    Will S Member

    Messages:
    717
    Joined:
    Nov 30, 2004
    Location:
    Madison, Wis
    Shooter:
    8x10 Format
    I thought that robots.txt just worked all on its lonesome and that the bot read it and processed accordingly. These must be bots that aren't adhering to the standard? Or did someone remove your files?

    Good luck with that. I once ran a site in the early 90s that kicked off large database queries in a lot of the links. We got brought down by Google for about a week. And this was before we had any means of telling the spider to stop! We had to call Google and ask them to quit it, and they wouldn't kill their jobs once they had started, etc., etc. Thankfully the robots.txt method showed up shortly thereafter (this was in 1994).

    Best,

    Will
     
  11. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Hi Will, yeah robots.txt seems quite useless these days..
     
  12. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    *UPDATE*

    Hello all, rather than leave the site down, I have made it so registered users have access. This means searchbots & spambots which are categorized as "guests" will not be able to attach to us... which means we should be ok. I'll keep an eye on things. Unfortunately the day job is calling so hopefully this will keep us going until I can resolve the other issues. Thanks
     
  13. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Might be ok now

    The robot blocking application is back in action, and I'm shocked to see it has blocked about 75 Inktomi Slurp spiders from our forums! This is a Yahoo spider system. I like us being indexed on the internet but these search sites like Yahoo, google, msn, need to learn how to control their systems! Anyway, things look ok right now, time will tell. Thanks
     
  14. titrisol

    titrisol Member

    Messages:
    1,671
    Joined:
    Aug 2, 2004
    Location:
    Rotterdam
    Shooter:
    Multi Format
    Can the RSS feed have something to do with it?
    I added apug to My Yahoo! for one

     
  15. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    Nope, it's definitely search spiders. There are still some issues to work through so no guarantees we are 100% up. The robot control application is running in a limitted way until I can get hold of their support again..
     
  16. Sean

    Sean Admin Staff Member Admin

    Messages:
    9,339
    Joined:
    Aug 29, 2002
    Location:
    New Zealand
    Shooter:
    Multi Format
    *update..

    I hate to say all is well.. again but i did make progress in getting the spider blocking app fully functional. I noticed all the Inktomi Slurp spider listings showing in our guests list have dropped off. We'll see how it goes.
     
  17. Flotsam

    Flotsam Member

    Messages:
    3,221
    Joined:
    Sep 30, 2002
    Location:
    S.E. New Yor
    Did you really change servers again?