A very simple task booking system. It's just sql, jinja2 and python.
You can download a binary, generated with pyinstaller from the releases tab.
The first step is to create a database and populate it with the entries.
toad init tasks.db --template examples/motus_pipeline/motus_pipeline.jinja2 --entries-file examples/motus_pipeline/entries.txtThis command will create a tasks.db sqlite, this database will be used to keep track of the status of the tasks.
Using the example in examples/motus_pipeline/entries.txt.
TODO: improve the layout of this code snippet
$ sqlite3 tasks.db "select * from entry limit 10;"
created_at updated_at id name
-------------------------- -------------------------- ---------- ----------
2023-06-18 18:44:44.621508 2023-06-18 18:44:45.139609 1 SRR5787989
2023-06-18 18:44:44.621508 2023-06-18 18:44:45.358367 2 SRR5787990
2023-06-18 18:44:44.621508 2023-06-18 18:44:45.511141 3 SRR5787991
2023-06-18 18:44:44.621508 2023-06-18 18:44:45.774346 4 SRR5787992
2023-06-18 18:44:44.621508 2023-06-18 18:44:46.067799 5 SRR5787993
2023-06-18 18:44:44.621508 2023-06-18 18:44:46.279143 6 SRR5787994
2023-06-18 18:44:44.621508 2023-06-18 18:44:46.460734 7 SRR5787995
2023-06-18 18:44:44.621508 2023-06-18 18:44:46.703233 8 SRR5787996
2023-06-18 18:44:44.621508 2023-06-18 18:44:46.879553 9 SRR5787997
2023-06-18 18:44:44.621508 2023-06-18 18:44:47.152089 10 SRR5787998
$ sqlite3 toad_test.db "select * from task limit 10;"
created_at updated_at id entry_id template_id status task_launch_exitcode task_launch_stdout task_launch_stderr task_execution_exitcode task_execution_stdout task_execution_stderr
-------------------------- -------------------------- ---------- ---------- ----------- ---------- -------------------- ------------------ ------------------ ----------------------- --------------------- ---------------------
2023-06-18 18:44:44.621508 2023-06-18 18:44:54.943962 51 51 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:55.093358 52 52 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:55.244337 53 53 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:55.389798 54 54 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:55.818848 55 55 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:56.083998 56 56 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:56.293171 57 57 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:56.426117 58 58 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:56.552228 59 59 1 pending NULL NULL NULL NULL NULL NULL
2023-06-18 18:44:44.621508 2023-06-18 18:44:56.689957 60 60 1 pending NULL NULL NULL NULL NULL NULL
The task template needs to contain the self-contained bash command to be used to run the task. Toad doesn't assume anything, it will just merge the template with the entry name and will launch the instruction.
You can see an example in examples/motus_pipeline/motus_pipeline.jinja2.
To run tasks you need to run:
toad run toad_test.db [--batch-size]
In order to check the status of the jobs, the user needs to provide a python script to do it. The script needs to have a method with the following name and signature:
def check(entry_name: str, launch_exitcode: str, launch_stdout: str, launch_stderr: str):
"""Run this method to check if an entry task is either still running, succesfully completed or failed.
:entry_name: The entry to be checked
:launch_exitcode: The exit code when the task was launched
:launch_stdout: The stdout when the task was launched
:launch_stdout: The stderr when the task was launched
:return: [ bool, int, str, str] -> is the task stil running, exit code when checking, stdout and stderr of the command used to check.
"""
passYou can see an example in examples/motus_pipeline/motus_pipeline_check.py
To run the checks on jobs already running:
toad check toad_test.db --check-script check_script.pyRead the CONTRIBUTING.md file.