A collection of Python helper functions for use in ETL pipelines
pip install codema-dev-tasks
tasks:
- source: codema_dev_tasks.requests.fetch_file
name: download_YOUR_FILENAME
params:
url: YOUR-URL
product: YOUR-FILEPATH
host | use? | note |
---|---|---|
http(s) | url: https://www.WEBSITE.com |
- |
s3 | url: s3:https://BUCKET-NAME/OBJECT-NAME |
If the data is not public you must save your credentials as environmental variables in a .env file |
Add a .env
file with your credentials to your current working directory
MY_CREDENTIALS=*****
Load the .env
file within the fetch_file
task
tasks:
- source: codema_dev_tasks.requests.fetch_file
params:
url: YOUR-URL
dotenv_path: "{{here}}/.env"
product: YOUR-FILEPATH