It is a simple image crawler with Selenium, but its differentiation is that it can detect objects in an image with a customized pre-trained model (Yolov5) and improve the model's performance using crawled images.
# I used it to crawl anus diseases images.
Used Skills
- Cloud: AWS S3, EC2, RDS (PostgreSQL)- Container: Docker, Docekr Compose
- CI/CD: GitHub Action, AWS CodeDeploy
- Automation: Airflow
Install
Clone repo and install requirements.txt
git clone https://github.com/farmboy-dev/Anus # clone
pip install -r requirements.txt # installCollector
If the object is detected -> Crawl the image# at Anus
python ./yolov5/image_collector.pyVerifier
The customized pre-trained model is not perfect, so sometime it detects a wrong object. That is reason why I made the verifier to feed good quality images to the model.# at Anus
python ./yolov5/image_verifier.pyCropper
My purpose is to detect anus disease (object) and crop the object not whole image.# at Anus
python ./yolov5/image_cropper.py- Data Version Control
- Tracking a status of models (performace, reproducibility, etc)