Note: this repository is not actively maintained. You can find a newer version of the adapter with Athena Engine 2 support, seeds and more in the following repository:
This plugin can be installed via pip:
$ pip install git+https://github.com/Dandandan/dbt-athena.git
A dbt profile can be configured to run against Athena using the following configuration:
| Option | Description | Required? | Example |
|---|---|---|---|
| s3_staging_dir | The location where Athena stores meta info | Required | s3:https://bucket/staging |
| schema | Specify the schema (athena database) to build models into | Required | dev |
| database | Data catalog | Required | awsdatacatalog |
| region_name | Specify in which AWS region it should connect | Required | eu-west-1 |
| threads | How many threads dbt should use | Optional(default=1) |
8 |
| max_retry_number | Number for retries for exponential backoff | Optional(default=5) |
8 |
| max_retry_delay | Maximum delay for exponential backoff in seconds | Optional(default=100) |
8 |
Example profiles.yml entry:
athena:
target: athena
outputs:
athena:
type: athena
database: awsdatacatalog
schema: dev
region_name: eu-west-1
threads: 8
s3_staging_dir: s3:https://athena-staging-bucket/
Due to the nature of Athena, not all core dbt functionality is supported. The following features of dbt are not implemented on Athena:
- Archival
- Incremental models
If you are interested in helping to add support for this functionality in dbt on Athena, please open an issue!
Known issues:
- Quoting is not supported
- Install tox:
pip install tox- Run unit tests:
tox -e unitAt this time, an AWS account is not provided in order to run the tests in CI. We kindly ask contributors/reviewers to use their own AWS accounts in order to test contributions.
You can also reach out in the Slack #athena channel for someone to run the tests for you.
Steps:
- Clone the dbt-integration-tests repository
git clone --branch athena-support https://github.com/EarnestResearch/dbt-integration-tests.git- Run the tests:
Additionally, you might need to set the AWS_PROFILE environment variable if you use another value than "default" (for example if you connect to multiple AWS accounts or assume different IAM roles)
AWS_DEFAULT_REGION=us-west-2 ATHENA_S3_STAGING_DIR=s3:https://dbt-athena-integration-tests/tests/ DBT_PROFILES_DIR=$(pwd)/test/integration/ tox -e integration-athenaEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct.