80legs.com - scourge of crawlers
2012-05-05 07:21
[zl]
I just run today into http://www.80legs.com/webcrawler.html on customer's webserver. It crawled by many threads and way too high rate for a slow PHP webshop hosted on an old P4 Xeon/few RAM server.
A tried filtering by robots.txt first:
On the following 15 minutes there was no request to robots.txt at all. At this point I decided to create a more powerful filter at once. I put these lines to virtualhost:
Only one question left: why?
I will update here if I get some news from customer about his recent marketing actions. I am almost sure this plague was not for nothing.
A tried filtering by robots.txt first:
User-agent: 008
Disallow: /
On the following 15 minutes there was no request to robots.txt at all. At this point I decided to create a more powerful filter at once. I put these lines to virtualhost:
<Directory /var/www/homedir>
<Files*>
SetEnvIfNoCase User-Agent "www\.80legs\.com" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Files>
</Directory>
Only one question left: why?
I will update here if I get some news from customer about his recent marketing actions. I am almost sure this plague was not for nothing.
Netcraft stat for www.zl.hu
2008-03-07 13:40
[zl]
Graffiti Art
2008-03-03 15:30
[zl]
Birkebeinen Senter, Bergen, Norway.

Be happy
2008-02-10 19:55
[zl]
Some moss-creature. Blåmanen, Bergen, Norway.

Graffiti Art
2007-11-20 20:47
[zl]
Created by anonymous artist in Sjøgaten tunnel, Bergen, Norway.
