This project was inspired by Spider Trap
A simple tool for catching web crawlers.
cargo build --release
cargo run
cargo run -- --port 8080 --directories ./web-directories.txt --min-links 5 --max-links 10 --min-length 3 --max-length 20
Usage: spidertrap-rs [OPTIONS]
Options:
--port <PORT> [default: 8080]
--directories <DIRECTORIES>
--min-links <MIN_LINKS> [default: 5]
--max-links <MAX_LINKS> [default: 10]
--min-length <MIN_LENGTH> [default: 3]
--max-length <MAX_LENGTH> [default: 20]
-h, --help Print help