Yahoo Slurp is a flood machine!
Let me start with a simple question! Why should I pay monthly for a service that is useless? Actually I don’t pay Yahoo to crawl and show my websites into their stupid search engine (this is free) but I must say that I am paying for bandwidth and resources for my dedicated server. Here is the problem! I have two article directories, each one with at least 100000 free articles and it is normal that some web crawlers to crawl my websites, but Yahoo’s crawler is a bit insupportable because of the flood it does. Yahoo Slurp was present on my website every minute, every day, crawling all my articles with an entire class of ips, ips starting with 74.6.*.* and 67.195.*.* .
You have the results bellow, in a printscreen from my awstats. Compare Yahoo to Google, given the fact that Yahoo is not sending any traffic and Google is sending 70% of my traffic.
- Yahoo : 81069 Hits | 6.37 GB Bandwidth
- Google : 18126 Hits | 239.73 MB Bandwidth
The solution for this problem is simple. Ban Yahoo’s ips on your server using htaccess or by using Ip deny manager from your cpanel/plesk administration center.
I added the following code to my .htaccess file to ban Yahoo’s ips:
order allow,deny deny from 67.195. deny from 74.6. allow from all
You should take this in consideration if you are hosted on a shared plan, because cpu and memory resources are limited and after all Yahoo isn’t so important.
This article has 6 Comments
I’ve read some of your stuff and tho I believe you have some valid points on some of the issues you’re discussing, this is just stupid.
Tacking in consideration that I don’t receive any traffic from yahoo, I don’t see any problem
. Research a bit on digitalpoint’s forum and you’ll see that I am not the only one that has problems with yahoo, a lot of people complain on this issue 
you don’t receive any traffic… (yet?)
maybe you should stress the issue of WHY you’re not getting traffic (duplicate articles maybe?), rather then just ban the bots. Yahoo is not google, true, but the traffic they’re sending is NOT negligible.
look: it’s your site of course and you’re welcome to do whatever you think it’s best… and although I’m not saying you don’t have your (valid?) reasons to ban the bots, I’m not sure it’s a good idea to advise others to do the same… at least not without making sure they completely understand exactly WHAT is going on – on that respect, you have a title that says “yahoo floods” and then the most preeminent part of your article is the htaccess code and that could lead to un-wanted results. Maybe change the article title to something like “yahoo slurp CAN flood CERTAIN sites” would be better
additionally, maybe “buy decent hosting” would be just as good of an advice as well
I agree with your article, for some reason yahoo slurp continually goes through 1 site we are hosting overloading the database with countless queries to the database. 24 hrs a day 7 days a week. It has a large database with over 56k entries. The site is an off the shelf product so it may need looking at but why would a bot continually spider a site. Thanks for the post at least I know it is not just us getting smashed by the yahoo bot.
All the other bots no problems. And it is the same IPs as you reported.
I have tried setting the robots.txt to slow it down but to no avail. I do not like banning it but it is creating problems for that site.
Hi
I totally agree with you as for my site 99.99 % are coming from Google
No one needs yahoo They wants Money for sending visitors .
I banned them via robots.txt
Thanks
I have exactly the same problem with Yahoo on two particular sites. It seems the moment a site delays sending a response, showing it’s struggling to keep up with demand, Yahoo just keeps requesting rather than backing off a bit. Additionally, it seems to group IP addresses together so that I will see up to 10 different Yahoo addresses crawling my server and up to 18 connections from a single IP address. As far as I am concerned, it’s the equivalent of a DDOS attack as they are not abiding by the robots.txt entries I set.
Pings:
Leave a Reply