ScrapydWeb: Full-featured web UI for Scrapyd cluster management, Scrapy log analysis and visualization
-
Scrapyd Cluster Management
- Group, filter and select any numbers of nodes
- Execute command on multinodes with one click
-
Scrapy Log Analysis
- Stats collection
- Progress visualization
- Logs categorization
-
All Scrapyd API Supported
- Deploy project, Run Spider, Stop job
- List projects/versions/spiders/running_jobs
- Delete version/project
-
Others
- Basic auth for web UI
- Accessing Scrapyd servers protected by basic auth
To install ScrapydWeb, simply use pip:
pip install scrapydweb
- Run
scrapydweb -h
to get help, and a config file named scrapydweb_settings.py would be copied to the working directory, then you can custom config with it. - Run
scrapydweb
- Visit https://127.0.0.1:5000 (It's recommended to use Google Chrome to get better experience.)