Ever heard of googlebot causing a forum to run slow?

[ZacFields]

Check this out guys,

I've been having speed problems for a couple days on my forums. I couldn't figure out for the life of me what was up. I noticed though, there was an IP address that had been on my site all day long that was registering under the WHOIS as GoogleBot.

I banned the IP address and suddenly my site is running faster. Has anyone ever heard of anything like this?

I'm going to post the IP up here so anyone can confirm if this is really a googlebot or an imposter. The ip is 66.249.66.210 (not sure if that's legal, Mods you can delete it if it's a problem). Like I said, banned this IP and the site seems to be running fine now.

Zac

[Teelk]

Yup, I've definitely heard of this. I've also heard of googlebot eating up a ton of monthly bandwidth, forcing webmasters to buy more.

One possible solution is to create a robots.txt, if you don't already have one, and place it in your forum root. Then add the following lines to tell googlebot to slow down.[code]User-agent]

You can also try going to ACP>General Admin>Meta Tags + and setting "META Revisit-after" to 15. Although, I've heard that most modern bots ignore meta tags, it can't hurt to try.

EDIT: I've also found this straight from google.com http://www.google.com/support/webmaster ... swer=48620

[MrGrim]

can someone type up one?

[ZacFields]

Thanks teelk I will give that a try.

I guess if people get on here complaining that their site has become slow... that might be something to check into. I don't know what the googlebot was doing that was causing this but definitely when I banned that IP my forums run as fast as ever now.

Also one other thing worth noting, I looked at my "Exploit Attempts" and checked some of those IP's and noticed quite a few of those banned IP's from exploit attempts were googlebots.

Zac

[ZacFields]

Yes I would have to confirm this did the trick.

At first I thought it was because I did a rebuild of the search database, but I think since the forums were disabled for about half an hour while I did that, the googlebot must have left temporarily. So when the forum came back on, initially it was fast but about 10 minutes later it started getting slow again.

This would be the FIRST thing I would suggest to someone who feels as though they are having a sudden problem with forum performance. This googlebot was causing my CPU usage to spike to 100% multiple times per minute and thus causing my pageloads to range anywhere from 5 seconds to 30 seconds.

In fact, I think I'm going to create another topic on here that will include all suggestions for speeding up a site, including this one. Always get into your admin panel and do a quick WHOIS of all the IP addresses that are not logged in. If you see any search engine bots, try a quick ban of that IP and see if that solves your speed issue...should do the trick instantly.

Look for that new topic soon, should help A LOT of people out as I suspect this may have been my problem all along. It would explain why when I transferred the domain name to my new VPS, suddenly my old shared host was running okay. Googlebot is on my site almost 24-7 and it is noteworthy that many MANY of the banned IP addresses in my "exploit attempts" section are googlebots. I have over 100 exploit attempts on my forum in the last 6 months...most of which I am noticing to be googlebot.

Somebody should fabricate a modification that would monitor how much resources are being consumed by each search engine bot. I knew googlebot was always on my site but I just never knew it was so resource-consuming.

Zac

[Michaelo]

On a similar note... I recently noticed my local server running very slow... I noticed ever time I opened a page it took fifteen to thirty seconds to display (previously it took one or two seconds)... I check the fire wall to see if something else was happening and I found firefox was connecting to Alexa (I must admit I don't know what Alexa is... some type of add on) any way I disallowed it from connect to Alexa and back came the speed....

Now all I need to do is find out what Alexa is <img>
Mike

[ZacFields]

ALexa is a near useless tool supposedly used to rank all the websites on the internet by traffic.

If you go to alexa.com and type in your web address you can see where you rank. The problem is that it only counts people who have the alexa toolbar so it's not exactly accurate. I don't think my site's even ranked on there and I get over 70,000 hits daily yet I know websites who get a lot less traffic than me that are rated pretty high on there.

Zzac

[Helter]

Alexa tool bar= spyware