Skip to content

lun-energy/installer-scrapers

Repository files navigation

StoreScraper

EDIT

You have to set the Mapbox API key in the settings.py file. It is at the top(MAPBOX_API_KEY)

Installing Python

  1. Download and install Python
    32-bit Windows:
    https://www.python.org/ftp/python/3.11.0/python-3.11.0.exe

    64-bit Windows:
    https://www.python.org/ftp/python/3.11.0/python-3.11.0-amd64.exe

    Make sure to check the "Add python.exe to PATH" at the bottom of the installation window.
    image

Installing the requirements

Open a command prompt and navigate to the scraper's directory.
The easiest way is to open the scraper's folder and type cmd in the address bar. image
Type the following command, you have to do this only once:

pip install -r requirements.txt

Starting the scraper

You can start the scraper by typing the following command:

python main.py

Multithreading

The scraper will run all stores in parallel.
However, in order to reduce the server load, each website will receive only one simultaneous request with 1 second cooldown.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages