Improve documentation

README.md

@@ -6,34 +6,40 @@ Shards database, collecting shard repositories, releases and dependencies.
 
 ### Prerequisites
 
-* Crystal 0.30.1
-* PostgresSQL database (for data storage).
-  Should probably work with all versions => 10. Tested with PostgreSQL 11.5
+* Crystal (0.30.1 or later)
+* PostgreSQL database (version 12)
+* [dbmate](https://github.com/amacneil/dbmate) for database migrations
 
+#### Database
 PostgreSQL databases needs to be created manually.
-Connection configuration is read from environment variables:
+
+Connection configuration is read from environment variables.
 
 * development database: `DATABASE_URL`
 * testing database: `TEST_DATABASE_URL` (only for running tests)
-* sentry: `SENTRY_DSN` (optional, for sending error reports to sentry)
 
-Install database schema by running `make db test_db`.
+Install database schema:
+
+* `make db` - development database
+* `make test_db` - test database
 
 ### Shard Dependencies
 
 Run `shards install` to install dependencies.
 
 ## Usage
 
-Run `shards build worker` to build the worker executable.
+Run `make bin/worker` to build the worker executable.
 
-* `bin/worker import_catalog`: Reads repositories mentioned in `catalog/*.yml` files and
-  synchronizes them with the database.
-* `bin/worker sync_repos [hours [ratio]]`: Synchronizes data from git repositories to database for all
-  repositories not synced in the last `hours` hours (default: `24`) or never synced at all. `ratio`
-  specifies the amount of outdated repositories being run in this job (default: `2.0 / hours`).
-  This enables an even distribution of batch sizes. Recommended to be run every hour.
+* `bin/worker import_catalog https://github.com/shardbox/catalog`:
+  Reads catalog description from the catalog repository and imports shards to the database.
+* `bin/worker sync_repos`: Synchronizes data from git repositories to database for all
+  repositories not synced in the last 24 hours or never synced at all.
+  Recommended to be run every hour.
 * `bin/worker updated_metrics`: Updates shard metrics in the database. Should be run once per day.
+* `bin/worker loop` starts a worker loop which schedules regular `sync_repos` and `update_metrics`
+  jobs. It also listens to notifications sent through PostgreSQL notify channels in order
+  to trigger catalog import.
 
 A web application for browsing the database is available at https://github.com/shardbox/shardbox-web
 
@@ -51,4 +57,4 @@ Run `make test` to execute the spec suite.
 
 ## Contributors
 
-- [Johannes Müller](https://github.com/straight-shoota) - creator and maintainer
+* [Johannes Müller](https://github.com/straight-shoota) - creator and maintainer

src/worker.cr

@@ -33,9 +33,12 @@ ENV["GIT_ASKPASS"] = "/usr/bin/test"
 def show_help(io)
   io.puts "shards-toolbox worker"
   io.puts "commands:"
-  io.puts "  import_catalog [path]        import catalog data from [path] (default: ./catalog)"
-  io.puts "  sync_repos [hours [ratio]]   syncs repos not updated in last [hours] ([ratio] 0.0-1.0)"
-  io.puts "  update_metrics               update shard metrics (should be run once per day)"
+  io.puts "  import_catalog [<path>]         import catalog data from <path> (default: ./catalog)"
+  io.puts "  sync_repos [<hours> [<ratio>]]  syncs repos not updated in last <hours> (<ratio> 0.0-1.0)"
+  io.puts "  sync_repo <repo_ref>            sync a single repo identified by <repo_ref>"
+  io.puts "  update_metrics                  update shard metrics (should be run once per day)"
+  io.puts "  loop                            run worker loop to schedule sync and update tasks"
+  io.puts "  help                            show usage instructions"
 end
 
 def sync_all_pending_repos