Enlighten me a little. What does the bot do that causes problems? Also, how will the data on APUG be searchable if you block the bots (or is that the price to pay for not permitting a bot to crawl the site)?
well the bots are like leeches. They find your site and latch onto it, sometimes for days pulling all the data from your pages. I think what's happening is that these bots might hit something they think should exist but doesn't. For example maybe a gallery image or a forum post, it gets stuck on this, glitches, and the cpu usage goes to 100%. People on dedicated servers with a lot of grunt can usually ride these things out, but we don't have that luxury yet. Most of the site is already indexed by the major engines so we still get some good visibility for now. When I have more time I'll try to get to the bottom of this and get some indexing back online..
FWIW, I just read an article in one of the IT mags that indicated some nefarious sites are using Google searches to probe web sites for security vulnerabilities. I wonder if this might be what's happening, Sean.
[COLOR=SlateGray]"You can't depend on your eyes if your imagination is out of focus." -Mark Twain[/COLOR]
is that these bots might hit something they think should exist but doesn't. For example maybe a gallery image or a forum post, it gets stuck on this, glitches, and the cpu usage goes to 100%.
This is something very much apart of the world of a few of the PHP kits. They do the silly assumption that if a resource does not exist then deliver a default resource.. and something this gets recursive.. One solution, if you can't fix it, is to deliver the pages via a reverse proxy server..