Added new profile mapping configuration for Teradata #1077

sc250072 · 2024-07-02T12:18:04Z

Description

Teradata has Provider in airflow and adapter in dbt. The cosmos library doesn't have profile configuration with mapping support. This PR address this issue.

Related Issue(s)

Closes #1053

Breaking Change?

No Breaking Change.

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

netlify · 2024-07-02T12:18:19Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`2bb37a1`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66850469680d0800080b3b39

cosmos/profiles/teradata/user_pass.py

tatiana

This looks great, @sc250072 , thank you very much for adding support to Teradata! I added some minor comments in-line

codecov · 2024-07-02T14:09:00Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.14%. Comparing base (30a7e5f) to head (2bb37a1).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1077      +/-   ##
==========================================
+ Coverage   96.12%   96.14%   +0.02%     
==========================================
  Files          62       63       +1     
  Lines        3249     3266      +17     
==========================================
+ Hits         3123     3140      +17     
  Misses        126      126

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cosmos/profiles/teradata/user_pass.py

1. Removed mock_profile as it inherits the parent mock_profile 2. Removed TYPE_CHECKING

sc250072 · 2024-07-02T14:51:28Z

This looks great, @sc250072 , thank you very much for adding support to Teradata! I added some minor comments in-line

Thank you @tatiana . Addressed the review comments.

tests/profiles/teradata/test_teradata_user_pass.py

tatiana

Thanks, @sc250072 for being so quick at addressing the comments. Happy for us to merge this PR once all the checks pass

sc250072 · 2024-07-03T06:04:04Z

Thanks, @sc250072 for being so quick at addressing the comments. Happy for us to merge this PR once all the checks pass

Thank you for the quick response on the PR. Right now, I'm conducting integration testing by manually installing the .whl file on my local machine. Could you please provide any documentation or guidance on implementing integration tests that can be run through GitHub workflows?

cosmos/profiles/teradata/user_pass.py

pankajastro · 2024-07-03T07:20:20Z

We have test file https://github.com/astronomer/astronomer-cosmos/blob/main/tests/test_example_dags.py which run some dags of this repo https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags also you can find some tests decorated with @pytest.mark.integration runs as integration tests. But running integration tests from this repo may require some Teradata credentials so I'm not sure if you have those at the moment

pankajastro

lgtm

Co-authored-by: Pankaj Singh <[email protected]>

tatiana · 2024-07-03T08:55:57Z

We have test file https://github.com/astronomer/astronomer-cosmos/blob/main/tests/test_example_dags.py which run some dags of this repo https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags also you can find some tests decorated with @pytest.mark.integration runs as integration tests. But running integration tests from this repo may require some Teradata credentials, so I'm not sure if you have those at the moment.

@sc250072 complemeting @pankajastro 's message: #1077 (comment)

Could the Teradata team share specific credentials to run integration tests from the Cosmos repo to validate that these tests pass? In that case, we'd be happy to remove the community flag.

As @pankajastro mentioned, we could either have an example DAG with Teradata or other tests marked with integration.

Another alternative, similar to Airflow, is to run a Cosmos DAG using this profile in your own CI/CD against the Cosmos main branch and share a dashboard/page with us with the job status (if you could extend https://teradata.github.io/airflow/. It would be awesome). In this case, you wouldn't have to share credentials with us, and we could help guarantee the stability of this feature. In this case, we could remove the is_community flag again.

tatiana · 2024-07-03T08:56:32Z

@sc250072 this work looks great, we'll be merging this PR and we can address removing the is_community in a separate PR

sc250072 · 2024-07-03T15:12:11Z

@sc250072 this work looks great, we'll be merging this PR and we can address removing the is_community in a separate PR

Thank you @tatiana .

sc250072 · 2024-07-04T05:04:08Z

@pankajastro @tatiana any expected date of new version of astronomer-cosmos, which includes teradata support. https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst

tatiana · 2024-07-04T11:26:52Z

@sc250072 it'll be released as part of 1.6, which is currently planned for the end of July.

Would you like us to create an alpha (e.g., 1.6.0a1) for you to validate the feature beforehand?

sc250072 · 2024-07-04T12:27:50Z

@sc250072 it'll be released as part of 1.6, which is currently planned for the end of July.

Would you like us to create an alpha (e.g., 1.6.0a1) for you to validate the feature beforehand?

Yes that would be help to validate feature before release.

tatiana · 2024-07-05T12:26:27Z

@sc250072 you can validate the feature in Cosmos 1.6.0a1

sc250072 · 2024-07-10T08:21:58Z

@tatiana I have tested Cosmos 1.6.0a1 and identified an issue with the Teradata mock profile, documented here: #1087. I have resolved this issue and submitted a pull request at #1088.

After addressing the issue, I validated the fix using a locally built .whl file and tested it with several Cosmos DAGs within the dbt-jaffle-shop project, utilizing the dbt-teradata adapter. Everything is functioning correctly now.

Could we proceed with another alpha release incorporating these changes?"

tatiana · 2024-07-10T12:24:52Z

Hi @sc250072 , thanks for all the work on this!

Yes, we'll be releasing a new alpha this week Nd we'll let you know

Teradata has [Provider](https://airflow.apache.org/docs/apache-airflow-providers-teradata/stable/index.html) in airflow and [adapter](https://github.com/Teradata/dbt-teradata) in dbt. The cosmos library doesn't have profile configuration with mapping support. This PR address this issue. Closes: astronomer#1053

sc250072 · 2024-07-16T05:30:35Z

Hi @sc250072 , thanks for all the work on this!

Yes, we'll be releasing a new alpha this week Nd we'll let you know

@tatiana any update on new alpha release

@pankajkoti

New Features * Add support for loading manifest from cloud stores using Airflow Object Storage by @pankajkoti in #1109 * Cache ``package-lock.yml`` file by @pankajastro in #1086 * Support persisting the ``LoadMode.VIRTUALENV`` directory by @tatiana in #1079 * Add support to store and fetch ``dbt ls`` cache in remote stores by @pankajkoti in #1147 * Add default source nodes rendering by @arojasb3 in #1107 * Add Teradata ``ProfileMapping`` by @sc250072 in #1077 Enhancements * Add ``DatabricksOauthProfileMapping`` profile by @CorsettiS in #1091 * Use ``dbt ls`` as the default parser when ``profile_config`` is provided by @pankajastro in #1101 * Add task owner to dbt operators by @wornjs in #1082 * Extend Cosmos custom selector to support + when using paths and tags by @mvictoria in #1150 * Simplify logging by @dwreeves in #1108 Bug fixes * Fix Teradata ``ProfileMapping`` target invalid issue by @sc250072 in #1088 * Fix empty tag in case of custom parser by @pankajastro in #1100 * Fix ``dbt deps`` of ``LoadMode.DBT_LS`` should use ``ProjectConfig.dbt_vars`` by @tatiana in #1114 * Fix import handling by lazy loading hooks introduced in PR #1109 by @dwreeves in #1132 * Fix Airflow 2.10 regression and add Airflow 2.10 in test matrix by @pankajastro in #1162 Docs * Fix typo in azure-container-instance docs by @pankajastro in #1106 * Use Airflow trademark as it has been registered by @pankajastro in #1105 Others * Run some example DAGs in Kubernetes execution mode in CI by @pankajastro in #1127 * Install requirements.txt by default during dev env spin up by @@CorsettiS in #1099 * Remove ``DbtGraph.current_version`` dead code by @tatiana in #1111 * Disable test for Airflow-2.5 and Python-3.11 combination in CI by @pankajastro in #1124 * Pre-commit hook updates in #1074, #1113, #1125, #1144, #1154, #1167 --------- Co-authored-by: Pankaj Koti <[email protected]> Co-authored-by: Pankaj Singh <[email protected]>

satish-chinthanippu added 2 commits July 1, 2024 06:01

Implemented new profile mapping class for teradata

640719b

Added profile mapping for teradata

e574457

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jul 2, 2024

sc250072 temporarily deployed to external July 2, 2024 12:18 — with GitHub Actions Inactive

dosubot bot added the area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc label Jul 2, 2024

sc250072 mentioned this pull request Jul 2, 2024

Add support for dbt-teradata #1053

Closed

1 task

tatiana reviewed Jul 2, 2024

View reviewed changes

cosmos/profiles/teradata/user_pass.py Outdated Show resolved Hide resolved

tatiana reviewed Jul 2, 2024

View reviewed changes

cosmos/profiles/teradata/user_pass.py Show resolved Hide resolved

tatiana reviewed Jul 2, 2024

View reviewed changes

cosmos/profiles/teradata/user_pass.py Outdated Show resolved Hide resolved

Added is_community and removed unwanted code

b0410a3

1. Removed mock_profile as it inherits the parent mock_profile 2. Removed TYPE_CHECKING

sc250072 temporarily deployed to external July 2, 2024 14:49 — with GitHub Actions Inactive

tatiana reviewed Jul 2, 2024

View reviewed changes

tests/profiles/teradata/test_teradata_user_pass.py Show resolved Hide resolved

tatiana approved these changes Jul 2, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jul 2, 2024

Merge branch 'main' into teradata-profile

5b91419

sc250072 had a problem deploying to external July 3, 2024 05:29 — with GitHub Actions Error

pankajastro reviewed Jul 3, 2024

View reviewed changes

cosmos/profiles/teradata/user_pass.py Outdated Show resolved Hide resolved

pankajastro approved these changes Jul 3, 2024

View reviewed changes

Removed None as method doesn't return None from filter_null

2bb37a1

Co-authored-by: Pankaj Singh <[email protected]>

sc250072 temporarily deployed to external July 3, 2024 07:57 — with GitHub Actions Inactive

tatiana merged commit 1c9e1f5 into astronomer:main Jul 3, 2024
64 checks passed

tatiana added this to the Cosmos 1.6.0 milestone Jul 4, 2024

tatiana mentioned this pull request Jul 5, 2024

Release 1.6.0 #1080

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added new profile mapping configuration for Teradata #1077

Added new profile mapping configuration for Teradata #1077

sc250072 commented Jul 2, 2024

netlify bot commented Jul 2, 2024 •

edited

Loading

tatiana left a comment

codecov bot commented Jul 2, 2024 •

edited

Loading

sc250072 commented Jul 2, 2024

tatiana left a comment

sc250072 commented Jul 3, 2024

pankajastro commented Jul 3, 2024 •

edited

Loading

pankajastro left a comment

tatiana commented Jul 3, 2024 •

edited

Loading

tatiana commented Jul 3, 2024

sc250072 commented Jul 3, 2024

sc250072 commented Jul 4, 2024

tatiana commented Jul 4, 2024

sc250072 commented Jul 4, 2024

tatiana commented Jul 5, 2024

sc250072 commented Jul 10, 2024

tatiana commented Jul 10, 2024

sc250072 commented Jul 16, 2024

Added new profile mapping configuration for Teradata #1077

Added new profile mapping configuration for Teradata #1077

Conversation

sc250072 commented Jul 2, 2024

Description

Related Issue(s)

Breaking Change?

Checklist

netlify bot commented Jul 2, 2024 • edited Loading

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

tatiana left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 2, 2024 • edited Loading

Codecov Report

sc250072 commented Jul 2, 2024

tatiana left a comment

Choose a reason for hiding this comment

sc250072 commented Jul 3, 2024

pankajastro commented Jul 3, 2024 • edited Loading

pankajastro left a comment

Choose a reason for hiding this comment

tatiana commented Jul 3, 2024 • edited Loading

tatiana commented Jul 3, 2024

sc250072 commented Jul 3, 2024

sc250072 commented Jul 4, 2024

tatiana commented Jul 4, 2024

sc250072 commented Jul 4, 2024

tatiana commented Jul 5, 2024

sc250072 commented Jul 10, 2024

tatiana commented Jul 10, 2024

sc250072 commented Jul 16, 2024

netlify bot commented Jul 2, 2024 •

edited

Loading

codecov bot commented Jul 2, 2024 •

edited

Loading

pankajastro commented Jul 3, 2024 •

edited

Loading

tatiana commented Jul 3, 2024 •

edited

Loading