The Mwmbl crawler is a distributed crawler with a central server. The central server collects batches of crawled data from the clients and stores it in long term storage for indexing and analysis.

Clone this repository git clone https://github.com/mwmbl/crawler-server then enter cd crawler-server To start crawler server via docker first edit .env file and then run docker-compose up -d To rebuild image execute docker-compose build