Skip to content

🏘 A RESTful API for housing website with Elastic Stack and MongoDB

License

Notifications You must be signed in to change notification settings

Sirius207/HousesAPI

Repository files navigation

HousesAPI

A RESTful API for housing website, built with secure Elastic Stack and MongoDB.

Build Status Known Vulnerabilities Maintainability codecov FOSSA Status Code style: black Conventional Commits pre-commit

Features

  • housing data Crawler
  • RESTful API with MongoDB for house data querying and creating
  • RESTful API with Token Authorization
  • API Doc built with Aglio
  • Elasticsearch Cluster Storage
  • House Data with Kibana Display
  • Filebeat Log Dashboard for MongoDB
  • APM Dashboard for API application
  • Metricbeat Dashboard for MongoDB

Documents

Getting Started

Prerequisites

  • python ^3.7.3
  • docker ^18.09.2
  • docker-compose ^1.17.1
  • Tesseract OCR ^4.x

Running Development

a. Install pipenv environment

make init
pipenv shell

b. Activate MongoDB

cd docker-manifests
docker-compose -f docker-mongo.yml up -d

c. Activate Elasticsearch, Kibana

  1. add .env to environment/

  2. setup elasticsearch Account at first usage

cd docker-manifests
docker-compose -f create-certs.yml run --rm create_certs
docker-compose -f docker-elastic.yml up -d
docker exec es01 /bin/bash -c "bin/elasticsearch-setup-passwords auto --batch --url https://es01:9200"
docker-compose -f docker-elastic.yml down
  1. setup ELASTICSEARCH_PASSWORD in .env

  2. copy es01.crt & kib01.crt to your computer and set certificate

docker cp es01:/usr/share/elasticsearch/config/certificates/es01/es01.crt ./
docker cp es01:/usr/share/elasticsearch/config/certificates/kib01/kib01.crt  ./
  1. restart elasticsearch & kibana
docker-compose  -f docker-elastic.yml up -d

d. Activate Filebeat, APM, MetricBeat

  1. start filebeat, apm, metricbeat
cd docker-manifests
docker-compose -f docker-beats.yml up -d
  1. setup filebeat
docker exec -it filebeat01 bash
filebeat setup
filebeat -e
  1. setup apm
docker exec -it apm01 bash
apm-server -e
  1. setup metricbeat
docker exec -it metricbeat01 bash
metricbeat01 setup
metricbeat01 -e

e. Execute Crawler

parse house data in Taipei (city_id=1) and Export House data to csv

cd services
export PYTHONPATH=$PYTHONPATH:$PWD

python crawler/main.py \
    --urls_file data/urls.csv \
    --data_file data/temp_info.csv\
    --city_id 1 \
    --url_start 0 \
    --url_end 250

f. Import Data to Database

  1. Import house data to MongoDB (from csv)
py api/loader/csv_to_mongo.py --file data/temp_info.csv
  1. Import house data to Elasticsearch (from csv)
py api/loader/csv_to_es.py --file data/temp_info.csv

g. Start RESTful API

python api/app.py

Running Production

Run Crawler, API, Nginx Proxy

docker-compose -f docker-house.yml up -d

Just Run API

Build API Docker Image

docker build . -f Dockerfile.crawler -t house-api-img --no-cache

Start WSGI in Container

docker run  --env-file .env --name house-api -d -p 5000:5000 house-api-img

Just Run Crawler

Build Crawler Docker Image

docker build . -f Dockerfile.crawler -t house-crawler-img --no-cache

Start Crawler in Container

docker run  --env-file .env --name house-crawler  house-crawler-img

Running the tests

Installing Dev Packages

pipenv install --dev

Unit Test

Insert Fake Data to MongoDB

cd services
export PYTHONPATH=$PYTHONPATH:$PWD
python api/loader/csv_to_mongo.py --file api/tests/api/fake_houses.csv

create default api account

python api/loader/init_account.py

Execute Tests

make tests

create API Doc (Use Aglio)

aglio --theme-variables streak  -i docs/api.apib --theme-template triple -o docs/api.html

Coding style tests

make lint

Check MongoDB Data

docker exec -it mongodb bash
mongo -u <user_name>
use <db>

Changelog

  • Check CHANGELOG.md

Contributing

  • Po-Chun, Lu

License

FOSSA Status