Scrapy spider to crawl and fetch information about real estate information
This project uses pipenv for package/dependency and virtualenv management, to learn more about how to use pipenv
Requirements:
brew install pipenv
pipenv --two
pipenv lock
pipenv installRedis cluster for local dev (Celery):
brew install redis
brew services start redisPostgres cluster for local dev (Backend database):
brew install postgres
brew services start postgresrun the following command from root directory
cd services/craigslist
scrapy crawl craigbot_all -o craigslist_result.csvcd services
export POSTGRES_DB_URI="postgres:https://postgres@localhost:5432/helios"; python bots.pyexport FLASK_ENV=development
export POSTGRES_DB_URI="postgres:https://postgres@localhost:5432/helios"
FLASK_APP=app.py flask run --debugger
# to trigger the bot, run
curl https://localhost:5000export POSTGRES_DB_URI="postgres:https://postgres@localhost:5432/helios"
python app.py
# connect to database
psql postgres -U postgres
\c helios
# execute create_initial_schema.sql script
# To test postgres database CRUD operations
cd services/postgres
export POSTGRES_DB_URI="postgres:https://postgres@localhost:5432/helios"
python postgres.pyhelios system is set up to be in continous deployment to Heroku platform at https://dashboard.heroku.com/apps/beast-helios tracking master branch
# deploy to Heroku, just check in/ merge into master branch
# restart app
heroku restart
# set/unset environment variables
heroku config:set <env_var>
heroku config:unset <env_var>
# check logs
heroku logs --tailNote Commits are associated with GPG signing key