- Date: May 11,2008
- In: SEO
Yahoo Slurp is a flood machine!
Let me start with a simple question! Why should I pay monthly for a service that is useless? Actually I don’t pay Yahoo to crawl and show my websites into their stupid search engine (this is free) but I must say that I am paying for bandwidth and resources for my dedicated server. Here is the problem! I have two article directories, each one with at least 100000 free articles and it is normal that some web crawlers to crawl my websites, but Yahoo’s crawler is a bit insupportable because of the flood it does. Yahoo Slurp was present on my website every minute, every day, crawling all my articles with an entire class of ips, ips starting with 74.6.*.* and 67.195.*.* .
You have the results bellow, in a printscreen from my awstats. Compare Yahoo to Google, given the fact that Yahoo is not sending any traffic and Google is sending 70% of my traffic.
- Yahoo : 81069 Hits | 6.37 GB Bandwidth
- Google : 18126 Hits | 239.73 MB Bandwidth
The solution for this problem is simple. Ban Yahoo’s ips on your server using htaccess or by using Ip deny manager from your cpanel/plesk administration center.
I added the following code to my .htaccess file to ban Yahoo’s ips:
order allow,deny deny from 67.195. deny from 74.6. allow from all
You should take this in consideration if you are hosted on a shared plan, because cpu and memory resources are limited and after all Yahoo isn’t so important.
3 Responses to “Yahoo Slurp is a flood machine!”
ionut
May 18th, 2008 at 5:05 am |
I’ve read some of your stuff and tho I believe you have some valid points on some of the issues you’re discussing, this is just stupid.
admin
May 18th, 2008 at 6:52 am |
Tacking in consideration that I don’t receive any traffic from yahoo, I don’t see any problem
. Research a bit on digitalpoint’s forum and you’ll see that I am not the only one that has problems with yahoo, a lot of people complain on this issue 
ionut
May 18th, 2008 at 7:33 am |
you don’t receive any traffic… (yet?)
maybe you should stress the issue of WHY you’re not getting traffic (duplicate articles maybe?), rather then just ban the bots. Yahoo is not google, true, but the traffic they’re sending is NOT negligible.
look: it’s your site of course and you’re welcome to do whatever you think it’s best… and although I’m not saying you don’t have your (valid?) reasons to ban the bots, I’m not sure it’s a good idea to advise others to do the same… at least not without making sure they completely understand exactly WHAT is going on - on that respect, you have a title that says “yahoo floods” and then the most preeminent part of your article is the htaccess code and that could lead to un-wanted results. Maybe change the article title to something like “yahoo slurp CAN flood CERTAIN sites” would be better
additionally, maybe “buy decent hosting” would be just as good of an advice as well
Leave a reply