Comments on: Yahoo Slurp is a flood machine! http://www.acosmin.com/yahoo-slurp-is-a-flood-machine/ Blogging, internet marketing and premium wordpress themes Sat, 31 Jul 2010 00:04:13 +0000 hourly 1 http://wordpress.org/?v=3.0.1 By: salubrium http://www.acosmin.com/yahoo-slurp-is-a-flood-machine/comment-page-1/#comment-40549 salubrium Fri, 01 Jan 2010 11:12:05 +0000 http://www.acosmin.com/?p=192#comment-40549 I have exactly the same problem with Yahoo on two particular sites. It seems the moment a site delays sending a response, showing it's struggling to keep up with demand, Yahoo just keeps requesting rather than backing off a bit. Additionally, it seems to group IP addresses together so that I will see up to 10 different Yahoo addresses crawling my server and up to 18 connections from a single IP address. As far as I am concerned, it's the equivalent of a DDOS attack as they are not abiding by the robots.txt entries I set. I have exactly the same problem with Yahoo on two particular sites. It seems the moment a site delays sending a response, showing it’s struggling to keep up with demand, Yahoo just keeps requesting rather than backing off a bit. Additionally, it seems to group IP addresses together so that I will see up to 10 different Yahoo addresses crawling my server and up to 18 connections from a single IP address. As far as I am concerned, it’s the equivalent of a DDOS attack as they are not abiding by the robots.txt entries I set.

]]> By: Carlos Guevara http://www.acosmin.com/yahoo-slurp-is-a-flood-machine/comment-page-1/#comment-23014 Carlos Guevara Wed, 27 May 2009 06:08:58 +0000 http://www.acosmin.com/?p=192#comment-23014 Hi I totally agree with you as for my site 99.99 % are coming from Google No one needs yahoo They wants Money for sending visitors . I banned them via robots.txt Thanks Hi
I totally agree with you as for my site 99.99 % are coming from Google
No one needs yahoo They wants Money for sending visitors .
I banned them via robots.txt
Thanks

]]> By: roger http://www.acosmin.com/yahoo-slurp-is-a-flood-machine/comment-page-1/#comment-6681 roger Tue, 14 Oct 2008 03:53:55 +0000 http://www.acosmin.com/?p=192#comment-6681 I agree with your article, for some reason yahoo slurp continually goes through 1 site we are hosting overloading the database with countless queries to the database. 24 hrs a day 7 days a week. It has a large database with over 56k entries. The site is an off the shelf product so it may need looking at but why would a bot continually spider a site. Thanks for the post at least I know it is not just us getting smashed by the yahoo bot. All the other bots no problems. And it is the same IPs as you reported. I have tried setting the robots.txt to slow it down but to no avail. I do not like banning it but it is creating problems for that site. I agree with your article, for some reason yahoo slurp continually goes through 1 site we are hosting overloading the database with countless queries to the database. 24 hrs a day 7 days a week. It has a large database with over 56k entries. The site is an off the shelf product so it may need looking at but why would a bot continually spider a site. Thanks for the post at least I know it is not just us getting smashed by the yahoo bot.
All the other bots no problems. And it is the same IPs as you reported.
I have tried setting the robots.txt to slow it down but to no avail. I do not like banning it but it is creating problems for that site.

]]> By: ionut http://www.acosmin.com/yahoo-slurp-is-a-flood-machine/comment-page-1/#comment-2552 ionut Sun, 18 May 2008 12:33:44 +0000 http://www.acosmin.com/?p=192#comment-2552 you don't receive any traffic... (yet?) maybe you should stress the issue of WHY you're not getting traffic (duplicate articles maybe?), rather then just ban the bots. Yahoo is not google, true, but the traffic they're sending is NOT negligible. look: it's your site of course and you're welcome to do whatever you think it's best... and although I'm not saying you don't have your (valid?) reasons to ban the bots, I'm not sure it's a good idea to advise others to do the same... at least not without making sure they completely understand exactly WHAT is going on - on that respect, you have a title that says "yahoo floods" and then the most preeminent part of your article is the htaccess code and that could lead to un-wanted results. Maybe change the article title to something like "yahoo slurp CAN flood CERTAIN sites" would be better :) additionally, maybe "buy decent hosting" would be just as good of an advice as well :) you don’t receive any traffic… (yet?)
maybe you should stress the issue of WHY you’re not getting traffic (duplicate articles maybe?), rather then just ban the bots. Yahoo is not google, true, but the traffic they’re sending is NOT negligible.

look: it’s your site of course and you’re welcome to do whatever you think it’s best… and although I’m not saying you don’t have your (valid?) reasons to ban the bots, I’m not sure it’s a good idea to advise others to do the same… at least not without making sure they completely understand exactly WHAT is going on – on that respect, you have a title that says “yahoo floods” and then the most preeminent part of your article is the htaccess code and that could lead to un-wanted results. Maybe change the article title to something like “yahoo slurp CAN flood CERTAIN sites” would be better :)

additionally, maybe “buy decent hosting” would be just as good of an advice as well :)

]]> By: admin http://www.acosmin.com/yahoo-slurp-is-a-flood-machine/comment-page-1/#comment-2551 admin Sun, 18 May 2008 11:52:42 +0000 http://www.acosmin.com/?p=192#comment-2551 Tacking in consideration that I don’t receive any traffic from yahoo, I don’t see any problem :) . Research a bit on digitalpoint’s forum and you’ll see that I am not the only one that has problems with yahoo, a lot of people complain on this issue :P Tacking in consideration that I don’t receive any traffic from yahoo, I don’t see any problem :) . Research a bit on digitalpoint’s forum and you’ll see that I am not the only one that has problems with yahoo, a lot of people complain on this issue :P

]]> By: ionut http://www.acosmin.com/yahoo-slurp-is-a-flood-machine/comment-page-1/#comment-2550 ionut Sun, 18 May 2008 10:05:37 +0000 http://www.acosmin.com/?p=192#comment-2550 I've read some of your stuff and tho I believe you have some valid points on some of the issues you're discussing, this is just stupid. I’ve read some of your stuff and tho I believe you have some valid points on some of the issues you’re discussing, this is just stupid.

]]>