This project was designed to create a universal web scraper that traverses through links of a provided web page and scrapes all the texts using Breadth-First technique
To put our spider to work, go to the project’s top level directory and run:
cd webscraper/spiders/
scrapy crawl amazing