Prefect (https://docs.prefect.io/) Task Flow paradigm base code for data science research
pip install -r requirements.txt
pip install -r requirements-dev.txt
Example test:
ENV_FOR_DYNACONF=test nosetests tests/tasks/sample_task/sample_task_test.py
In order for the cache mechanism to work, run the following:
export PREFECT__FLOWS__CHECKPOINTING=true
or add it to .bashrc:
echo 'export PREFECT__FLOWS__CHECKPOINTING=true' >> ~/.bashrc
Running the example flow:
python -m sample.flows.sample_flow