COPY THE GLOBALBLACKLIST.CONF FILE FROM THE REPO $APACHE_CONF is generally located at /etc/apache2 or /etc/httpd depending on OS
Copy the contents of globalblacklist.conf into your $APACHE_CONF/custom.d folder. e.g. /etc/apache2/custom.d on Ubuntu/Debian /etc/httpd on RHEL/centos You need to create this folder.
The following directions use /etc/apache2 as an example.
sudo mkdir /etc/apache2/custom.d
sudo wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/globalblacklist.conf -O /etc/apache2/custom.d/globalblacklist.conf
If your Linux distribution does not have wget you can replace the wget commands using curl as follows:
curl -sL https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/globalblacklist.conf -o /etc/apache2/custom.d/globalblacklist.conf
WHITELIST ALL YOUR OWN DOMAIN NAMES AND IP ADDRESSES
Whitelist all your own domain names and IP addresses. Please note important changes, this is now done using include files so that you do not have to keep reinserting your whitelisted domains and IP addresses every time you update.
❗ ❗ ❗ WARNING: Please do NOT ever use 127.0.0.1 as a whitelisted IP address in whitelist-ips.conf it will cause the blocker to stop functioning.
- copy the whitelist-ips.conf file into that folder
sudo wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/whitelist-ips.conf -O /etc/apache2/custom.d/whitelist-ips.conf
- copy the whitelist-domains.conf file into the same folder
sudo wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/whitelist-domains.conf -O /etc/apache2/custom.d/whitelist-domains.conf
Use nano, vim or any other text editor to edit both whitelist-ips.conf and whitelist-domains.conf to include all your own domain names and IP addresses that you want to specifically whitelist from the blocker script.
When pulling any future updates now you can simply pull the latest globalblacklist.conf file and it will automatically include your whitelisted domains and IP addresses. No more remembering having to do this yourself.
DOWNLOAD CUSTOM BLACKLIST INCLUDE FILE FOR IP ADDRESS AND IP RANGE BLOCKING
Blacklist any IP addresses or Ranges you wish to keep out of your servers. Please note important changes, this is now done using include files so that you have full control over what IP addresses and IP Ranges and blocked from your Apache Server.
- copy the blacklist-ips.conf file into that folder
sudo wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/blacklist-ips.conf -O /etc/apache2/custom.d/blacklist-ips.conf
Use nano, vim or any other text editor to edit the blacklist-ips.conf file as you like.
When pulling any future updates now your custom IP blacklist will not be overwritten.
DOWNLOAD CUSTOM BAD REFERRER WORDS INCLUDE FILE FOR CUSTOMIZED SCANNING OF BAD WORDS
Scan for any bad referrer words you wish to keep out of your servers. Please note important changes, this is now done using include files so that you have full control over what IP addresses and IP Ranges and blocked from your Apache Server.
- copy the bad-referrer-words.conf file into that folder
sudo wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/bad-referrer-words.conf -O /etc/apache2/custom.d/bad-referrer-words.conf
Use nano, vim or any other text editor to edit the bad-referrer-words.conf file as you like.
When pulling any future updates now your custom bad referrer words list will not be overwritten.
DOWNLOAD CUSTOM BLACKLIST USER-AGENTS INCLUDE FILE FOR CUSTOMIZED BLOCKING OF USER AGENTS
Allows you to add your own custom list of user agents with this new include file.
- copy the blacklist-user-agents.conf file into that folder
sudo wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/blacklist-user-agents.conf -O /etc/apache2/custom.d/blacklist-user-agents.conf
NOTE: On the Apache Blocker if you want to over-ride any of the whitelisted bots you can add them to this include file and the previously whitelisted bots in the blocker will be over-ridden by this include file. So let's say for some "obscure" reason you really do not want any search engines like Googlebot or Bingbot to ever access or index your site, you add them to your blacklist-user-agents.conf and they will be over-ridden from the earlier whitelisting in the blocker. This now gives users total control over the blocker without every having to try and modify the globalblacklist.conf file. So now you can customize all your include files and you can still pull the daily updates of globalblacklist.conf and it will not touch any of your custom include files.
INCLUDE THE GLOBALBLACKLIST.CONF
Include the globalblacklist.conf file in the beginning of a directory block just after your opening Options statements and before the rest of your host config example below. Remove the "<<<<<< This needs to be added" part
<VirtualHost *:80>
ServerName local.dev
ServerAlias www.local.dev
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Directory "/var/www/html">
AllowOverride All
Options FollowSymLinks
Include custom.d/globalblacklist.conf
</Directory>
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
You can include globalblacklist.conf globally (for all virtual hosts) if you put the following configuration after virtual host configuration.
# ######################################
# GLOBAL! deny bad bots and IP addresses
# ######################################
#
# should be set after <VirtualHost>s see https://httpd.apache.org/docs/2.4/sections.html#merging
<Location "/">
# AND-combine with preceding configuration sections
AuthMerging And
# include black list
Include custom.d/globalblacklist.conf
</Location>
TEST YOUR APACHE CONFIGURATION
Do an Apache2 Config Test
sudo apache2ctl configtest
If you get no errors then you followed my instructions so now you can make the blocker go live with a simple.
sudo service apache2 reload
or
sudo service httpd reload
The blocker is now active and working so now you can run some simple tests from another linux machine to make sure it's working.
TESTING*
Run the following commands one by one from a terminal on another linux machine against your own domain name. substitute yourdomain.com in the examples below with your REAL domain name
curl -A "googlebot" http:https://yourdomain.com
Should respond with 200 OK
curl -A "80legs" http:https://yourdomain.com
curl -A "masscan" http:https://yourdomain.com
Should respond with 403 Forbidden
curl -I http:https://yourdomain.com -e http:https://100dollars-seo.com
curl -I http:https://yourdomain.com -e http:https://zx6.ru
Should respond with 403 Forbidden
The Apache Ultimate Bot Blocker is now WORKING and PROTECTING your web sites !!!
UPDATING THE APACHE BAD BOT BLOCKER is now easy thanks to the automatic includes for whitelisting your own domain names.
Updating to the latest version is now as simple as:
sudo wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/globalblacklist.conf -O /etc/apache2/custom.d/globalblacklist.conf
sudo apache2ctl configtest
sudo service apache2 reload
And you will be up to date with all your whitelisted domains included automatically for you now.
See the latest auto updater bash script for Apache 2.2 and 2.4 contributed by Luke Taylor @lutaylor at:
Relax now and sleep better at night knowing your site is telling all those baddies FORBIDDEN !!!
With great thanks and appreciation to https://blog.nexcess.net/2011/01/21/one-liners-for-apache-log-files/
To monitor your top referer's for a web site's log file's on a daily basis use the following simple cron jobs which will email you a list of top referer's / user agents every morning from a particular web site's log files. This is an example for just one cron job for one site. Set up multiple one's for each one you want to monitor. Here is a cron that runs at 8am every morning and emails me the stripped down log of referers. When I say stripped down, the domain of the site and other referers like Google and Bing are stripped from the results. Of course you must change the log file name, domain name and your email address in the examples below. The second cron for collecting User agents does not do any stripping out of any referers but you can add that functionality if you like copying the awk statement !~ from the first example.
00 08 * * * tail -10000 /var/log/apache/mydomain-access.log | awk '$11 !~ /google|bing|yahoo|yandex|mywebsite.com/' | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -1000 | mail -s "Top 1000 Referers for Mydomain.com" [email protected]
00 08 * * * tail -50000 /var/log/apache/mydomain-access.log | awk '{print $12}' | tr -d '"' | sort | uniq -c | sort -rn | head -1000 | mail -s "Top 1000 Agents for Mydomain.com" [email protected]
Thousands of hours of programming and testing have gone into this project, show some love
Copyright (c) 2017 Mitchell Krog - [email protected]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.