An ecosystem of crawlers for detecting: leaks, sensitive data exposure and attempts exfiltration of data
This project is summarized in several crawlers that constitute a single ecosystem, that monitor certain channels such as: Github, Bing, Pastebin and iHaveBeenPwned? in order to perform data leak detection, exposed sensitive files and data exfiltration attempts.
- You can click here to see a diagram of the database and also a catalog on them.
# Download
$ git clone https://github.com/GouveaHeitor/uranus && cd uranus
# Building and starting MariaDB Database
$ docker build -t uranus-database ./rest-server/database/
$ docker run -d -p 3306:3306 --name database -e MARIADB_ROOT_PASSWORD=mypassword uranus-database
# Building and starting the REST API
$ docker build -t uranus-rest-server ./rest-server.
$ docker run -d -p 80:80 --name rest-server uranus-rest-server
# Building all crawlers/workers containers
$ docker build -t bing-crawler ./crawlers/bing/
$ docker build -t email-notify ./workers/email-notify
# Running all crawlers/workers containers
$ docker run -d --name bing bing-crawler
$ docker run -d --name email-notify email-notifyYour contributions and suggestions are heartily ♥ welcome. See here the contribution guidelines. Please, report bugs via issues page and for security issues, see here the security policy. (✿ ◕‿◕)
This work is licensed under MIT License.