This part of mds4all
is archived for now as it is built on Heroku who cancelled their free tier in 2022.
Minimalistic and free modern data stack, hence for all.
-
- Warehousing: Google BigQuery
- Orchestrating: GitHub Actions
- Ingesting: Meltano
- Transforming: dbt
- Cataloging: dbt docs on GitHub Pages
-
other
:- Code-editing: VS Code with:
Visualisation layer of mds4all
.
# Set up basics
$ pip install apache-superset
$ pip install psycopg2
$ pip install pybigquery # for BigQuery connection
$ pip freeze > requirements.txt # or just limit to installed above
$ echo "web: superset db upgrade" > Procfile
$ echo "python-3.8.2" > runtime.txt
# add configs to superset_config.py
# Generate secret key
$ python
>>> from cryptography import fernet
>>> fernet.Fernet.generate_key()
# Set up Heroku app
$ heroku login
$ heroku create mds4all-superset
$ heroku config:set SECRET_KEY=<SECRET_KEY>
$ heroku config:set PORT=<PORT>
$ heroku addons:create heroku-postgresql:hobby-dev
$ git push heroku main
$ heroku logs –-tail # check for errors
# Initialise Superset
$ heroku run bash
$ export FLASK_APP=superset
$ superset fab create-admin
$ superset init
$ exit
# Finalise app
$ echo 'web: gunicorn "superset.app:create_app()"' > Procfile
$ git push heroku main
$ heroku logs --tail # check for errors
# Open app
$ heroku open
- Sleep after 30 mins of inactivity
- 512 MB RAM
- 4 workers
- Gunicorn server
- No cache (like Redis)
- Database storage with 10K rows and 20 MB