Portal Home > Knowledgebase > Articles Database > Bot Agent majestic12.co.uk banging my server - how to fix
Bot Agent majestic12.co.uk banging my server - how to fix
| Posted by 111111z, 08-26-2008, 02:51 PM |
| This bot is a distributed application bot and started noticing a lot of traffic in my access log /var/log/apache2/access.log file
to search for it do
tail -n 100000 /var/log/apache2/access.log | grep "majestic12.co.uk"
this looks through the last 100000 lines of the access log file
for majestic12.co.uk
To stop this I put this code in my apache conf file
SetEnvIfNoCase User-Agent "majestic12.co.uk" stay_out
Order Allow,Deny
Deny from env=stay_out
Allow from all
and then reload apache
now when that bot trys to access any of my websites it gets a 403
error.
I figure someone else may find this helpful.
|
| Posted by MikeDVB, 08-26-2008, 03:02 PM |
| They'll still use bandwidth (although minimal)
<< Not needed any more >>
Last edited by writespeak; 08-27-2008 at 12:33 PM.
|
| Posted by FS - Mike, 08-26-2008, 03:40 PM |
| Good post, thanks for that info. I'll have to keep an eye on our logs to see if we get anything like that.
Mike
|
| Posted by vibrokatana, 08-26-2008, 03:48 PM |
| I've found that the bots sometimes try to exploit dynamically generated pages to try and overload the server. Hitting a slow generating page can bring down a server pretty quickly, particularly if they aren't using any caching.
Last edited by writespeak; 08-27-2008 at 12:35 PM.
Reason: Edited quoted portion of post to match edited quote
|
| Posted by 111111z, 08-26-2008, 03:59 PM |
| vibro
thats exactly what was happening....
I started looking at the logs because every other day the webserver
stopped responding and I had to restart apache.
once I started blocking the bot it stopped freezing so often....
|
| Posted by jzukerman, 08-26-2008, 04:05 PM |
| Are these bots coming from multiple IPs or just one? A simple iptables rule could block out a single IP address or even multiple if you can narrow down the list of suspected bots.
|
| Posted by 111111z, 08-26-2008, 04:07 PM |
| this bot is a distributed application so it could be any ip address
unfortunately.
|
| Posted by elvis1, 09-13-2009, 02:40 AM |
| watch it ! most likely not belonging to the project.. many fake bots have been doing so in its name
http://www.majestic12.co.uk/forum/vi...light=iptables
|
| Posted by plumsauce, 09-13-2009, 04:02 AM |
| The owner of the real bot is well known.
The bot is designed to respect robots.txt, so go to the site, find out what entry you need in robots.txt
That will stop the real one, anything else coming through is a fake.
|
| Posted by elvis1, 09-13-2009, 10:24 AM |
| thanks for the clarification with add data I did not know . I really like the project. So even though it was an old thread, I would not like mj12 to be trashed
|
Add to Favourites 
Print this Article
Also Read
rsync (Views: 468)
