funny utp cable
  en [magyar]
80legs.com - scourge of crawlers
2012-05-05 07:21 [ZL]
I just run today into http://www.80legs.com/webcrawler.html on customer's webserver. It crawled by many threads and way too high rate for a slow PHP webshop hosted on an old P4 Xeon/few RAM server.
A tried filtering by robots.txt first:
User-agent: 008
Disallow: /

On the following 15 minutes there was no request to robots.txt at all. At this point I decided to create a more powerful filter at once. I put these lines to virtualhost:
<Directory /var/www/homedir>
 <Files*>
  SetEnvIfNoCase User-Agent "www\.80legs\.com" bad_bot
  Order Allow,Deny
  Allow from all
  Deny from env=bad_bot
 </Files>
</Directory>

Only one question left: why?

I will update here if I get some news from customer about his recent marketing actions. I am almost sure this plague was not for nothing.
Netcraft stat for www.zl.hu
2008-03-07 13:40 [ZL]
Graffiti Art
2008-03-03 15:30 [ZL]
Birkebeinen Senter, Bergen, Norway. mino mano
Be happy
2008-02-10 19:55 [ZL]
Some moss-creature. Blåmanen, Bergen, Norway. blamanen_fjell
Graffiti Art
2007-11-20 20:47 [ZL]
Created by anonymous artist in Sjøgaten tunnel, Bergen, Norway. sjogaten_tunnel





















 

NEWSGROUPS
  Projects
  Support
more info: lyuk@zl.hu
FOR CUSTOMERS
OF OURSELVES
  Bit Vader
  Blog