A collection of Python helper functions for use in ETL pipelines
pip install codema-dev-taskstasks:
- source: codema_dev_tasks.requests.fetch_file
name: download_YOUR_FILENAME
params:
url: YOUR-URL
product: YOUR-FILEPATH| host | use? | note |
|---|---|---|
| http(s) | url: https://www.WEBSITE.com |
- |
| s3 | url: s3:https://BUCKET-NAME/OBJECT-NAME |
If the data is not public you must save your credentials as environmental variables in a .env file |
Add a .env file with your credentials to your current working directory
MY_CREDENTIALS=*****
Load the .env file within the fetch_file task
tasks:
- source: codema_dev_tasks.requests.fetch_file
params:
url: YOUR-URL
dotenv_path: "{{here}}/.env"
product: YOUR-FILEPATH