This repository contains sources of server part of Wroclive iOS app.

(Psst… the app itself is also open-sourced!)

We use Google Cloud for hosting.

We extensively use their free tier, to keep our hosting costs low. The only thing that we actually pay for is storage for App Engine instances, but this is about 0.20 zł (~$0.05) per month (only the instances in US are free and we chose Frankfurt because of GDPR and latency).

Anyway, this is how it works:

And this is the push notification pipeline:

App Engine

• Handles all traffic to wroclive.app and wroclive.app/api
• Runs code from AppEngine directory
• Every 1h it fetches MPK data (for example available lines and stop locations) from Firestore
• Every 5 seconds it fetches new vehicle locations from wroclaw.pl/open-data
• If you want to deploy it yourself (make deploy), then remember to put GCP-Credentials.json for app-engine-firestore service account in AppEngine directory
• Since AppEngine instances should not have state (they may be restarted by Google at whim) every 30s we store last heading/angle update location for every vehicle in the database. If something goes wrong we will use those locations to calculate new headings. This is also nice because now we can update the app without worrying about losing the state.

Firestore

• Stores persistent data:
  • lines - filled by ComputeEngine-Updater
  • stops - filled by ComputeEngine-Updater
  • notifications - filled by ComputeEngine-Notifications
  • push notification tokens - filled by AppEngine
  • send push notifications - filled by ComputeEngine-Notifications

Compute Engine

• Single instance named backend
• Startup script: /home/USERNAME/on-vm-startup.sh
• Runs code from following directories:
  • ComputeEngine
    • Manage ComputeEngine instance (installations, updates etc.)
    • Update flow:
      • make package to create .zip package with all of the ComputeEngine modules
      • make upload to send package to GCP
      • make connect to log into GCP
      • (on VM) ./install-package.sh
      • exit
      • make restart to restart VM (if needed)
  • ComputeEngine-Updater
    • Simple app that downloads GTFS file from wroclaw.pl/open-data and upload it to Firestore
    • Uses sqlite.org for intermediate processing
    • Before installing remember to put GCP-Credentials.json for compute-engine-firestore service account in ComputeEngine-Updater directory
  • ComputeEngine-PubSub
    • Pub/Sub subscriber
    • After receiving backend-update-gtfs-data message runs ComputeEngine-Updater
    • Starts when the Compute Engine instance starts
    • Before installing remember to put GCP-Credentials.json for pubsub service account in ComputeEngine-PubSub directory
  • ComputeEngine-Notifications
    • Gets latest tweets from @AlertMPK and sends them as push notifications
    • Stores notifications in Firestore
    • Starts when the Compute Engine instance starts
    • Before installing remember to put following in ComputeEngine-Notifications directory:
      • GCP-Credentials.json for compute-engine-firestore service account
      • Twitter-Credentials.json - see Twitter-Credentials-example.json for details
      • APN-Key.p8 - key for Apple Push Notification service
      • APN-Credentials.json - see APN-Credentials-example.json for details

Cloud Functions

• Runs code from CloudFunctions directory
• backendStart function
  • Looks for backend instance on Compute Engine and starts it
  • Scheduled by Cloud Scheduler
• backendStop function
  • Looks for backend instance on Compute Engine and stops it
  • scheduled by Cloud Scheduler

Pub/Sub - following topics need to be created:

• backend-stop
• backend-start
• backend-update-gtfs-data

Cloud Scheduler

• Every day at 1am (0 3 * * *): publish backend-stop message on Pub/Sub
• Every day at 5am (15 3 * * *): publish backend-start message on Pub/Sub
• Every day at 3am (30 3 * * *): publish backend-update-gtfs-data message on Pub/Sub

IAM & Admin - following service accounts are used:

• App Engine default service account

  • automatically created by App Engine
  • no configuration needed

• app-engine-firestore - account used by App Engine to read/write data in Firestore

  • Role: Viewer, Editor
  • Key should be exported and placed in AppEngine

• compute-engine-firestore - account used in ComputeEngine-Updater and ComputeEngine-Notifications to read/write data in Firestore

  • Role: Owner
  • Key should be exported and placed in ComputeEngine-Updater and ComputeEngine-Notifications

• pubsub - account used for creating subscriptions in ComputeEngine-PubSub

  • Roles: Editor, Pub/Sub Publisher, Pub/Sub Subscriber
  • Key should be exported and placed in ComputeEngine-PubSub

Monitoring

• SSL certificate expiring soon
  • Alert: 7 days prior
• Uptime check on wroclive.app/api (this will also keep our App Engine instance alive)
  • Protocol: HTTPS
  • Hostname: wroclive.app
  • Path: api
  • Check frequency: 10 minutes
  • Regions: Europe

Cloud Logging

• We use winstonjs/winston with @google-cloud/logging-winston backend

• Add following filers:

  • App Engine life cycle
    • Query: resource.type="gae_app" AND (textPayload=~"(Starting app|Quitting on terminated signal)$" OR protoPayload.methodName="google.appengine.v1.Versions.CreateVersion")
  • Api by user agent
    • Query: protoPayload.userAgent =~ "Wroclive.*"

• Log parser

  • Parse logs from Cloud Logging and output them in human-readable form

Error reporting - standard reporting by mail

• siege

  • preset to Unix siege tool + baseline - used to check burst performance
  • tiny module written in Python to check sustained performance

• postman.json - configuration file to import into postman app (it contains our endpoints + some basic tests)

Wroclive is licensed under the Mozilla Public License 2.0 license. See LICENSE for more information.