The robots.txt file residing in this directory should be used in addition to using the Nginx Bad Bot Blocker.

There is good reason for this.

It is considered good practice to actually tell a bot whether it is allowed or denied access.

So as a webmaster or server operator it is the right and also responsible thing to tell any of our listed bad bots to go away first.

If a bot ignores robots.txt (which many do) it will then be blocked outright by the blocker, make no mistake about that.

• Created by: https://github.com/mitchellkrogza for use on Nginx Web Server https://www.nginx.com/
• Copyright Mitchell Krog [email protected]

Not Using Nginx? See the

This robots.txt file is designed primarily for wordpress sites. This is revealed in the starting lines.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This means all user agents are ALLOWED to access all parts of the web sites by default

• they are NOT allowed to crawl the Wordpress /wp-admin/ folder
• they ARE allowed to access /wp-admin/admin-ajax.php

You can modify these beginning lines to your liking.

Then after the beginning lines we start explicitly telling bots to go away.

Example:

User-agent: 360Spider
Disallow:/
User-agent: 404checker
Disallow:/
.....
.....
.....
.....

Check out the updaterobots.sh bash script for a simple updater for all your web sites.

• https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker
• https://github.com/mitchellkrogza/Badd-Boyz-Hosts
• https://github.com/mitchellkrogza/Stop.Google.Analytics.Ghost.Spam.HOWTO
• https://github.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites
• https://github.com/mitchellkrogza/fail2ban-useful-scripts
• https://github.com/mitchellkrogza/linux-server-administration-scripts
• https://github.com/mitchellkrogza/Travis-CI-Nginx-for-Testing-Nginx-Configuration
• https://github.com/mitchellkrogza/Travis-CI-for-Apache-For-Testing-Apache-and-PHP-Configurations
• https://github.com/mitchellkrogza/Fail2Ban-Blacklist-JAIL-for-Repeat-Offenders-with-Perma-Extended-Banning
• https://github.com/funilrys/funceble

Come drop by and visit me at https://mitchellkrog.com or

Many Thanks to those contributing to this project.

Many parts of the generator scripts and code running behind this project have been adapted from snippets from hundreds of sources. In fact it is so hard to mention everyone but here are a few key people whose little snippets of code have helped me introduce new features all the time. Show them some love and check out some of their projects too.

• Stuart Cardall - https://github.com/itoffshore (Install, Update and Setup Scripts & Alpine Linux Package Maintainer)
• Nissar Chababy - https://github.com/funilrys/funceble (Excellent script for checking ACTIVE, INACTIVE and EXPIRED Domain Names)
• Marius Voila https://github.com/mariusv
• Cătălin Mariș https://github.com/alrra
• deformhead https://github.com/deformhead
• bluedragonz https://github.com/bluedragonz
• Alexander https://github.com/shoonois
• Steven Black https://github.com/StevenBlack
• Fail2Ban - https://github.com/fail2ban
• Stevie-Ray Hartog https://github.com/Stevie-Ray
• Sir Athos from StackOverFlow - http:https://stackoverflow.com/users/2245910/sir-athos (help with Travis Build Tagging and Committing)
• StackOverflow - http:https://stackoverflow.com/ (bash scripts from hundreds of questions and answers)
• SuperUser - http:https://superuser.com/ (snippets from various questions and answers)

If you believe your name should be here, drop me a line.

Thousand of hours of programming and testing have gone into this project, show some love

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.