Skip to content

Commit

Permalink
add man pages
Browse files Browse the repository at this point in the history 
with these installed in /usr/share/man/man1 you will be
able to run from a terminal:

man nbbb.config
man nbbb.example
man nbbb.readme
man nbbb.sample
  • Loading branch information
@itoffshore
itoffshore committed Mar 25, 2017
1 parent 41797e4 commit cbe529e
Show file tree
Hide file tree
Showing 4 changed files with 986 additions and 0 deletions.
203 changes: 203 additions & 0 deletions man/man1/nbbb.config.1
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
.TH "nbbb.config" 1 "23rd March 2017" "version: 2.2017.05" "INSTRUCTIONS"
.SH CONFIGURATION OF THE NGINX BAD BOT BLOCKER:
PLEASE READ CONFIGURATION INSTRUCTIONS BELOW THOROUGHLY
Created by: \[la]https://github.com/mitchellkrogza\[ra]
Copyright Mitchell Krog \[la][email protected]\[ra]
Version 2.2017.05
.PP
\fBIf you miss one step you will get an nginx EMERG error. This is normally a result of not downloading either blockbots.conf, ddos.conf, whitelist\-ips.conf, whitelist\-domains.conf or blacklist\-user\-agents.conf into your /etc/nginx/bots.d folder. If any of the include files are missing Nginx will EMERG and will not reload.\fP
.SH AUTO INSTALLATION INSTRUCTIONS
.PP
To Make Sure you copy all the correct files you can now use a simple bash setup script for copying the files into the correct nginx folders for you:
See: \[la]https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/installnginxblocker.sh\[ra]
.PP
\fBPlease Note:\fP the bash installer script does not carry out STEP 7 of the manual configuration instructions for you. YOU MUST edit any vhosts files yourself and manually add the entries in STEP 7 or the blocker will not actually be protecting any sites.
.SH MANUAL INSTALLATION INSTRUCTIONS
.SH STEP 1:
.PP
\fBCOPY THE GLOBALBLACKLIST.CONF FILE FROM THE REPO\fP
.PP
Copy the contents of \fB/conf.d/globalblacklist.conf\fP into your /etc/nginx/conf.d folder.
.PP
\fB\fCcd /etc/nginx/conf.d\fR
.PP
\fB\fCsudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx\-ultimate\-bad\-bot\-blocker/master/conf.d/globalblacklist.conf \-O globalblacklist.conf\fR
.SH STEP 2:
.PP
\fBCOPY THE INCLUDE FILES FROM THE REPO\fP
.RS
.IP \(bu 2
From your command line in Linux type
.RE
.PP
\fB\fCsudo mkdir /etc/nginx/bots.d\fR
.PP
\fB\fCcd /etc/nginx/bots.d\fR
.RS
.IP \(bu 2
copy the blockbots.conf file into that folder
.RE
.PP
\fB\fCsudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx\-ultimate\-bad\-bot\-blocker/master/bots.d/blockbots.conf \-O blockbots.conf\fR
.RS
.IP \(bu 2
copy the ddos.conf file into the same folder
.RE
.PP
\fB\fCsudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx\-ultimate\-bad\-bot\-blocker/master/bots.d/ddos.conf \-O ddos.conf\fR
.SH STEP 3:
.PP
\fBWHITELIST ALL YOUR OWN DOMAIN NAMES AND IP ADDRESSES\fP
.PP
Whitelist all your own domain names and IP addresses. \fBPlease note important changes\fP, this is now done using include files so that you do not have to keep reinserting your whitelisted domains and IP addresses every time you update.
.PP
\fB\fCcd /etc/nginx/bots.d\fR
.RS
.IP \(bu 2
copy the whitelist\-ips.conf file into that folder
.RE
.PP
\fB\fCsudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx\-ultimate\-bad\-bot\-blocker/master/bots.d/whitelist\-ips.conf \-O whitelist\-ips.conf\fR
.RS
.IP \(bu 2
copy the whitelist\-domains.conf file into the same folder
.RE
.PP
\fB\fCsudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx\-ultimate\-bad\-bot\-blocker/master/bots.d/whitelist\-domains.conf \-O whitelist\-domains.conf\fR
.PP
Use nano, vim or any other text editor to edit both whitelist\-ips.conf and whitelist\-domains.conf to include all your own domain names and IP addresses that you want to specifically whitelist from the blocker script.
.PP
When pulling any future updates now you can simply pull the latest globalblacklist.conf file and it will automatically include your whitelisted domains and IP addresses.
.SH STEP 4:
.PP
\fBBLACKLIST USING YOUR OWN CUSTOM USER\-AGENT BLACKLIST\fP
.PP
Copy the custom User\-Agents blacklist file into your /etc/nginx/bots.d folder
.PP
\fB\fCcd /etc/nginx/bots.d\fR
.RS
.IP \(bu 2
copy the blacklist\-user\-agents.conf file into the same folder
.RE
.PP
\fB\fCsudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx\-ultimate\-bad\-bot\-blocker/master/bots.d/blacklist\-user\-agents.conf \-O blacklist\-user\-agents.conf\fR
.PP
Use nano, vim or any other text editor to edit (if needed) blacklist\-user\-agents.conf to include your own custom list of bad agents that are not included in the blocker like "omgilibot" which some people choose to block.
.SH STEP 5:
.PP
\fBINCLUDE IMPORTANT SETTINGS IN NGINX.CONF\fP
\fBAlso see SAMPLE\-nginx.conf file in the root of this repository\fP
.PP
\fB\fCcd /etc/nginx/conf.d\fR
.RS
.IP \(bu 2
copy the botblocker\-nginx\-settings.conf file directly from the repo
.RE
.PP
\fB\fCsudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx\-ultimate\-bad\-bot\-blocker/master/conf.d/botblocker\-nginx\-settings.conf \-O botblocker\-nginx\-settings.conf\fR
.PP
\fBWhat is included in this settings file above for nginx?\fP
The important settings file above adds the rate limiting functions and hash_bucket settings for nginx for you. Below is what the file contains, you cn add these manually to your nginx.conf file if you so please but the include file above will do it for you ad nginx loads any .conf file in /etc/conf.d (See STEP 6)
.PP
.RS
server\fInames\fPhash\fIbucket\fPsize 64;
.PP
server\fInames\fPhash\fImax\fPsize 4096;
.PP
limit\fIreq\fPzone $binary\fIremote\fPaddr zone=flood:50m rate=90r/s;
.PP
limit\fIconn\fPzone $binary\fIremote\fPaddr zone=addr:50m;
.RE
.PP
\fBPLEASE NOTE:\fP The above rate limiting rules are for the DDOS filter, it may seem like high values to you but for wordpress sites with plugins and lots of images, it's not. This will not limit any real visitor to your Wordpress sites but it will immediately rate limit any aggressive bot. Remember that other bots and user agents are rate limited using a different rate limiting rule at the bottom of the globalblacklist.conf file.
.PP
The server\fInames\fPhash settings allows Nginx Server to load this very large list of domain names and IP addresses into memory. You can tweak these settings to your own requirements.
.SH STEP 6: \fBVERY IMPORTANT\fP
.PP
\fBMAKE SURE\fP that your nginx.conf file contains the following include directive. If it's commented out make sure to uncomment it or none of this will work.
.RS
.IP \(bu 2
\fB\fCinclude /etc/nginx/conf.d/*\fR
.RE
.SH STEP 7: \fBVERY IMPORTANT\fP
.PP
\fBADD INCLUDE FILES INTO A VHOST\fP
.PP
Open a site config file for Nginx (just one for now) and add the following lines.
.PP
\fBVERY IMPORTANT NOTE:\fP
.PP
These includes MUST be added within a \fBserver {}\fP block of a vhost otherwise you will get EMERG errors from Nginx.
.RS
.IP \(bu 2
\fB\fCinclude /etc/nginx/bots.d/blockbots.conf;\fR
.IP \(bu 2
\fB\fCinclude /etc/nginx/bots.d/ddos.conf;\fR
.RE
.SH STEP 8:
.PP
\fBTESTING YOUR NGINX CONFIGURATION\fP
.PP
\fB\fCsudo nginx \-t\fR
.PP
If you get no errors then you followed my instructions so now you can make the blocker go live with a simple.
.PP
\fB\fCsudo service nginx reload\fR
.PP
The blocker is now active and working so now you can run some simple tests from another linux machine to make sure it's working.
.SH STEP 9:
.PP
\fBTESTING\fP
.PP
Run the following commands one by one from a terminal on another linux machine against your own domain name.
\fBsubstitute yourdomain.com in the examples below with your REAL domain name\fP
.PP
\fB\fCcurl \-A "googlebot" http:https://yourdomain.com\fR
.PP
Should respond with 200 OK
.PP
\fB\fCcurl \-A "80legs" http:https://yourdomain.com\fR
.PP
\fB\fCcurl \-A "masscan" http:https://yourdomain.com\fR
.PP
Should respond with: curl: (52) Empty reply from server
.PP
\fB\fCcurl \-I http:https://yourdomain.com \-e http:https://100dollars\-seo.com\fR
.PP
\fB\fCcurl \-I http:https://yourdomain.com \-e http:https://zx6.ru\fR
.PP
Should respond with: curl: (52) Empty reply from server
.PP
The Nginx Ultimate Bot Blocker is now WORKING and PROTECTING your web sites !!!
.SH STEP 10:
.PP
\fBUPDATING THE NGINX BAD BOT BLOCKER\fP is now easy thanks to the automatic includes for whitelisting your own domain names.
.PP
Updating to the latest version is now as simple as:
.PP
\fB\fCcd /etc/nginx/conf.d\fR
.PP
\fB\fCsudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx\-ultimate\-bad\-bot\-blocker/master/conf.d/globalblacklist.conf\fR
.PP
\fB\fCsudo nginx \-t\fR
.PP
\fB\fCsudo service nginx reload\fR
.PP
\fBIn Alpine Linux you can run: /usr/sbin/updatenginxblocker\fP
.PP
And you will be up to date with all your whitelisted domains included automatically for you now.
.SH AUTO UPDATING:
.PP
See my latest auto updater bash script at:
.PP
\[la]https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/updatenginxblocker.sh\[ra]
.PP
Relax now and sleep better at night knowing your site is telling all those baddies they are FORBIDDEN !!!
.SH PULL REQUESTS:
.PP
To contribute your own bad referers please add them into the \[la]https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/Pull_Requests_Here_Please/badreferers.list\[ra] file and then send a Pull Request (PR).
.PP
\fBAll additions will be checked for accuracy before being merged.\fP
.SH ISSUES:
.PP
Log any issues regarding incorrect listings or any other problems on the issues system and they will be investigated and removed if necessary. I responde very quickly to user problems and have helped countless users for days on end to get their bot blocker working. You could say I am mad (disputable) but I love helping people and do not ignore issues or people with problems getting this to work.
145 changes: 145 additions & 0 deletions man/man1/nbbb.example.1
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
.TH "nbbb.example" 1 "23rd March 2017" "version: 2.2017.05" "Example SSL configuration"
.SH NGINX Example SSL configuration file for the NGINX Ultimate Bad Bot Blocker
using a Free SSL Certificate from Let's Encrypt
.PP
If this helps you You can buy me a beer \[la]https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BKF9XT6WHATLG\[ra] or send some cheese for my mouse \[la]https://www.gitcheese.com/app/#/projects/92bf5669-7d2c-447d-baa4-216ac9e720a6/pledges/create\[ra]
.PP
The sample NGINX configuration below is for an SSL site and includes the very important http (port 80) redirect to https (Port 443) which a lot of people tend to forget about. The configuration example below uses a Free SSL certificate from \[la]https://letsencrypt.org\[ra]
.PP
Make sure to test and reload nginx when you make changes. \fB\fCsudo nginx \-t\fR and if no errors then \fB\fCsudo service nginx reload\fR
.SH To Test Bad Referers
.PP
Then you must test running the following from the command line of another unix machine.
.PP
\fB\fCcurl \-I https://yourdomain.com \-e http:https://100dollars\-seo.com\fR
.PP
\fB\fCcurl \-I https://yourdomain.com \-e http:https://xxxrus.org\fR
.PP
\fB\fCcurl \-I https://yourdomain.com \-e https://100dollars\-seo.com\fR
.PP
\fB\fCcurl \-I https://yourdomain.com \-e https://sexobzor.info\fR
.PP
\fB\fCcurl \-I https://yourdomain.com \-e ftp:https://sexobzor.info\fR
.PP
You will get an empty reply meaning the Nginx Bad Bot Blocker is working. You will also notice if a bad referer comes from http:https://, https:// or even ftp:https:// it is blocked due to the special regex in this blocker which ignores whether it comes from http:https://, https:// or even ftp:https:// it is detected and BLOCKED !!!
.PP
Then try the following commands against your http site
.PP
\fB\fCcurl \-I http:https://yourdomain.com \-e http:https://100dollars\-seo.com\fR
.PP
\fB\fCcurl \-I http:https://yourdomain.com \-e http:https://xxxrus.org\fR
.PP
\fB\fCcurl \-I http:https://yourdomain.com \-e https://100dollars\-seo.com\fR
.PP
\fB\fCcurl \-I http:https://yourdomain.com \-e https://sexobzor.info\fR
.PP
You should see the response give you a 301 redirect:
.PP
.RS
.nf
HTTP/1.1 301 Moved Permanently
Location: https://yourdomain.com/
.fi
.RE
.PP
This means it is redirecting all http traffic (port 80) to https (port 443). At this point most bad bots and bad referrers give up and will not even bother to follow the redirect. If they do however they will get blocked.
.PP
\fBNOTE:\fP
I have overridden this behavior in the example below by also adding the include into the port80 site's configuration section before the Redirect conditions take effect. Which means bots and bad referers hitting your http site will get blocked and will not even be shown the redirect to your https site.
.SH To Test Bad User Agents
.PP
To test further, install User\-Agent Switcher for Chrome, set up a few bad bots like 80legs, masscan, AhrefsBot and switch to them while viewing your site in Chrome and you will see 403 Forbidden errors meaning the Nginx Bad Bot Blocker is working.
.PP
Or again using for those who love the command line. On another unix machine try some of these.
.PP
\fB\fCcurl \-A "80Legs" https://yourdomain.com\fR
.PP
\fB\fCcurl \-A "websucker" https://yourdomain.com\fR
.PP
\fB\fCcurl \-A "masscan" https://yourdomain.com\fR
.PP
\fB\fCcurl \-A "WeBsuCkEr" https://yourdomain.com\fR
.PP
\fB\fCcurl \-A "WeB suCkEr" https://yourdomain.com\fR
.PP
\fB\fCcurl \-A "Exabot" https://yourdomain.com\fR
.PP
You will get 403 forbidden responses on all of them meaning the Nginx Bad Bot Blocker is working 100%. You will also notice if a bot like websucker changes it's name to WeBsuCkEr it is detected regardless due to the wonderful case insensitive matching regex of this blocker. Test against any bot or referrer string in the bot blocker and you will always get a 403 forbidden.
.SH To Test Good User Agents
.PP
Try some of these from the command line of another unix machine and you will see that good bots specified in the Nginx Bad Bot blocker are granted access.
.PP
\fB\fCcurl \-A "GoogleBot" https://yourdomain.com\fR
.PP
\fB\fCcurl \-A "BingBot" https://yourdomain.com\fR
.PP
Now you can rest knowing your site is protected against over 4000 and growing bad bots and spam referrers and allowing all the good one's through.
.PP
Enjoy it and what this will do for your web site.
.SH Make sure to keep your /etc/conf.d/globalblacklist.conf file up to date
.PP
New referrers and bots are added every other day. Each time you update \fBMAKE SURE\fP to copy your whitelist section of IP addresses into the new file. A set of generator scripts are coming soon which will ease this burden for you allowing you to pull daily from the GIT repo and compile the scripts on your server automatically including your whitelisted IP's each time. These generator scripts are coming soon so please be patient as they have to be thoroughly tested for public use before I release them.
.PP
(See at very bottom of this page for all the Cloudflare IP ranges you should be whitelisting if you are on Cloudflare)
.SH EXAMPLE Nginx SSL site configuration file. (/etc/nginx/sites\-available/yourdomain.com")
.PP
.RS
.nf
server {
# SSL configuration
listen 443 ssl http2;
root /var/www/yourdomain.com;
server_name yourdomain.com www.yourdomain.com;
charset UTF\-8;
# Logging for the SSL version of our site
access_log /var/log/nginx/yourdomain.com\-access.log;
error_log /var/log/nginx/yourdomain.com\-error.log;

# SSL Configuration
# First include our certificates and chain of trust \- Using Let's Encrypt Free SSL
ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/yourdomain.com/chain.pem;
# Diffie\-Hellman parameter for DHE ciphersuites, recommended 2048 bits
ssl_dhparam /etc/nginx/ssl/dhparam.pem;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:128m;
ssl_session_tickets off;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# ciphers recommended by https://mozilla.github.io/server\-side\-tls/ssl\-config\-generator/
ssl_ciphers 'ECDHE\-ECDSA\-CHACHA20\-POLY1305:ECDHE\-RSA\-CHACHA20\-POLY1305:ECDHE\-ECDSA\-AES128\-GCM\-SHA256:ECDHE\-RSA\-AES128\-GCM\-SHA256:ECDHE\-ECDSA\-AES256\-GCM\-SHA384:ECDHE\-RSA\-AES256\-GCM\-SHA384:DHE\-RSA\-AES128\-GCM\-SHA256:DHE\-RSA\-AES256\-GCM\-SHA384:ECDHE\-ECDSA\-AES128\-SHA256:ECDHE\-RSA\-AES128\-SHA256:ECDHE\-ECDSA\-AES128\-SHA:ECDHE\-RSA\-AES256\-SHA384:ECDHE\-RSA\-AES128\-SHA:ECDHE\-ECDSA\-AES256\-SHA384:ECDHE\-ECDSA\-AES256\-SHA:ECDHE\-RSA\-AES256\-SHA:DHE\-RSA\-AES128\-SHA256:DHE\-RSA\-AES128\-SHA:DHE\-RSA\-AES256\-SHA256:DHE\-RSA\-AES256\-SHA:ECDHE\-ECDSA\-DES\-CBC3\-SHA:ECDHE\-RSA\-DES\-CBC3\-SHA:EDH\-RSA\-DES\-CBC3\-SHA:AES128\-GCM\-SHA256:AES256\-GCM\-SHA384:AES128\-SHA256:AES256\-SHA256:AES128\-SHA:AES256\-SHA:DES\-CBC3\-SHA:!DSS';
ssl_prefer_server_ciphers on;
add_header Strict\-Transport\-Security "max\-age=31536000; includeSubDomains";
ssl_stapling on;
ssl_stapling_verify on;

# Include our X\- Headers for Browser Cross\-Sniffing
add_header X\-Frame\-Options SAMEORIGIN;
add_header X\-Content\-Type\-Options nosniff;
add_header X\-XSS\-Protection "1; mode=block";


# ADD THE NGINX BAD BOT BLOCKER HERE (Please read full setup instructions)
include /etc/nginx/bots.d/blockbots.conf;
include /etc/nginx/bots.d/ddos.conf;

# Include Any Custom Configurations and Location Directives Here

# END OF SSL HOST CONFIG \- CLOSING BRACE BELOW THIS LINE
}
server {
# NOW WE REDIRECT ALL PORT 80 TRAFFIC TO PORT 443
listen 80;
server_name yourdomain.com www.yourdomain.com;
# Block Bad Bots even before they even get redirected
include /etc/nginx/bots.d/blockbots.conf;
include /etc/nginx/bots.d/ddos.conf;
return 301 https://yourdomain.com$request_uri;
# HAVE SEPARATE LOGGING FOR PORT 80 (otherwise use same log location as SSL site)
access_log /var/log/nginx/yourdomain.com\-80\-access.log;
error_log /var/log/nginx/yourdomain.com\-80\-error.log;
# END OF HTTP PORT 80 HOST CONFIG \- CLOSING BRACE BELOW THIS LINE
}
.fi
.RE
.SS If this helped you You can buy me a beer \[la]https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BKF9XT6WHATLG\[ra] or send some cheese for my mouse \[la]https://www.gitcheese.com/app/#/projects/92bf5669-7d2c-447d-baa4-216ac9e720a6/pledges/create\[ra]
Loading

0 comments on commit cbe529e

Please sign in to comment.