Web scraper utilizing scrapy to scrape live stock prices from the Nairobi Stock Exchange. The prices are then saved in Postgres Database after each scrape, we use sqlalchemy as ORM and psycopg2 as database connector.

The accompanying article can be found here

The actual platform we are scraping is afx website.

• Python and pip (I am currently using 3.9.2) Any version above 3.5 should work.

• An Africas Talking account.

  • Api Key and username from your account. Create an app and take note of the api key.

• Postgresql Database.

  • This could either be installed locally or via docker. This article is an awesome resource on how to get Postgresql and pgadmin4 installed as containers.

  Create a database nse_scraper . Either using SQL or 3-party client like pgadmin4 or dbeaver

Clone this repo

  git clone https://github.com/KenMwaura1/stock-price-scraper

Change into the directory

cd stock-price-scraper

Create a virtual environment (venv) to hold all the required dependencies.Here we use the built-in venv module.

python -m venv env

Activate the virtual environment

source env/bin/activate

Alternatively if you are using pyenv.

pyenv virtualenv nse_scraper
pyenv activate nse_scraper

Install the required dependencies:

pip install -r requirements

Change into the nse_scraper folder and create an environment file.

cd nse_scraper
touch .env 

Add your credentials as specified in the example file.

OR

Copy the provided example and edit as required:

cp .env-example env

Navigate up to the main folder stock-price-scraper Run the scraper and check the logs for any errors .

cd .. 
scrapy crawl afx_scraper

or Run the scraper and have it output to a json file to preview.

scrapy crawl afx_scraper -o test.json 

MIT