Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CI issues #1005

Merged
merged 4 commits into from
May 29, 2024
Merged

Fix CI issues #1005

merged 4 commits into from
May 29, 2024

Conversation

tatiana
Copy link
Collaborator

@tatiana tatiana commented May 28, 2024

Fix three problems observed while running the tests in the CI recently:

  1. Static check

Fix spelling issue captured by static check (#1000)

  1. Integration test that's no longer needed

The issue we were trying to capture for no longer happens in the latest version of the Apache Airflow provider apache-airflow-providers-postgres==5.11.1rc1: apache/airflow#39842

  1. Skip a buggy version of OL

There was a breaking change between openlineage-integration-common==1.14.0 and openlineage-integration-common==1.15.0 . It may have been an unintended side-effect of OpenLineage/OpenLineage#2693.

This is an example of how Cosmos was using DbtLocalArtifactProcessor - something that had been agreed upon in the past:

      openlineage_processor = DbtLocalArtifactProcessor(
            producer=OPENLINEAGE_PRODUCER,
            job_namespace=LINEAGE_NAMESPACE,
            project_dir=project_dir,
            profile_name=self.profile_config.profile_name,
            target=self.profile_config.target_name,
        )
        events = openlineage_processor.parse()
        for completed in events.completes:
            for output in getattr(completed, source):
                dataset_uri = output.namespace + "/" + output.name
                uris.append(dataset_uri)

In openlineage-integration-common==1.14.0 and earlier versions, this would create URIs in the format:

postgres:https://0.0.0.0:5432/postgres.public.stg_customers

Since openlineage-integration-common==1.15.0 , this leads to URIs being created in the format:

postgres.public.stg_customers/postgres:https://0.0.0.0:5432

This was fixed in OpenLineage/OpenLineage#2735 and released as part of OL 1.16:
https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0

captured by test

FAILED tests/operators/test_local.py::test_run_operator_dataset_emission_fails - Failed: DID NOT RAISE <class 'cosmos.exceptions.AirflowCompatibilityError'>
…st version of the Apache Airflow providers:

apache/airflow#39842

apache-airflow-providers-postgres==5.11.1rc1
Copy link

netlify bot commented May 28, 2024

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit c877e2a
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6655ffd47b15fd0008ee76da

Copy link

codecov bot commented May 28, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.65%. Comparing base (cb2a27a) to head (c877e2a).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1005      +/-   ##
==========================================
- Coverage   95.72%   95.65%   -0.07%     
==========================================
  Files          60       60              
  Lines        2926     2925       -1     
==========================================
- Hits         2801     2798       -3     
- Misses        125      127       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana marked this pull request as ready for review May 28, 2024 16:29
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. area:ci Related to CI, Github Actions, or other continuous integration tools area:dependencies Related to dependencies, like Python packages, library versions, etc profile:postgres Related to Postgres ProfileConfig labels May 28, 2024
"apache-airflow-providers-cncf-kubernetes>=5.1.1",
"apache-airflow-providers-amazon>=3.0.0,<8.20.0", # https://github.com/apache/airflow/issues/39103
"apache-airflow-providers-docker>=3.5.0",
"apache-airflow-providers-microsoft-azure",
"apache-airflow-providers-postgres",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bring back to alphabetic order

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label May 29, 2024
@tatiana tatiana merged commit 32c47ca into main May 29, 2024
62 of 63 checks passed
@tatiana tatiana deleted the fix-ci branch May 29, 2024 08:56
tatiana added a commit that referenced this pull request Jun 6, 2024
Fix three problems observed while running the tests in the CI recently:

1. Static check

Fix spelling issue captured by static check (#1000)

2. Integration test that's no longer needed

The issue we were trying to capture for no longer happens in the latest
version of the Apache Airflow provider
`apache-airflow-providers-postgres==5.11.1rc1`:
apache/airflow#39842

3. Skip a buggy version of OL

There was a breaking change between
`openlineage-integration-common==1.14.0` and
`openlineage-integration-common==1.15.0` . It may have been an
unintended side-effect of
OpenLineage/OpenLineage#2693.

This is an example of how Cosmos was using `DbtLocalArtifactProcessor` -
something that had been agreed upon in the past:

```
      openlineage_processor = DbtLocalArtifactProcessor(
            producer=OPENLINEAGE_PRODUCER,
            job_namespace=LINEAGE_NAMESPACE,
            project_dir=project_dir,
            profile_name=self.profile_config.profile_name,
            target=self.profile_config.target_name,
        )
        events = openlineage_processor.parse()
        for completed in events.completes:
            for output in getattr(completed, source):
                dataset_uri = output.namespace + "/" + output.name
                uris.append(dataset_uri)
```

In `openlineage-integration-common==1.14.0` and earlier versions, this
would create URIs in the format:
```
postgres:https://0.0.0.0:5432/postgres.public.stg_customers
```
Since openlineage-integration-common==1.15.0 , this leads to URIs being
created in the format:
```
postgres.public.stg_customers/postgres:https://0.0.0.0:5432
```

This was fixed in OpenLineage/OpenLineage#2735
and released as part of OL 1.16:
https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
@tatiana tatiana mentioned this pull request Jun 6, 2024
tatiana added a commit that referenced this pull request Jun 6, 2024
Bug fixes

* Fix the invocation mode for ``ExecutionMode.VIRTUALENV`` by @marco9663
in #1023
* Fix Cosmos ``enable_cache`` setting by @tatiana in #1025
* Make ``GoogleCloudServiceAccountDictProfileMapping`` dataset profile
arg optional by @oliverrmaa and @pankajastro in #839 and #1017
* Athena profile mapping set ``aws_session_token`` in profile only if it
exists by @pankajastro in #1022

Others

* Update dbt and Airflow conflicts matrix by @tatiana in #1026
* Enable Python 3.12 unittest by @pankajastro in #1018
* Improve error logging in ``DbtLocalBaseOperator`` by @davidsteinar in
#1004
* Add GitHub issue templates for bug reports and feature request by
@pankajkoti in #1009
* Add more fields in bug template to reduce turnaround in issue triaging
by @pankajkoti in #1027
* Fix ``dev/Dockerfile`` + Add ``uv pip install`` for faster build time
by @dwreeves in #997
* Drop support for Airflow 2.3 by @pankajkoti in #994
* Update Astro Runtime image by @RNHTTR in #988 and #989
* Enable ruff F linting by @pankajastro in #985
* Move Cosmos Airflow configuration to settings.py by @pankajastro in
#975
* Fix CI Issues by @tatiana in #1005
* Pre-commit hook updates in #1000, #1019

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Reeves <[email protected]>
Co-authored-by: Pankaj Koti <[email protected]>
Co-authored-by: davidsteinar <[email protected]>
Co-authored-by: David Steinar Asgrimsson <[email protected]>
Co-authored-by: Marco Yuen <[email protected]>
Co-authored-by: Pankaj Singh <[email protected]>
Co-authored-by: Ollie Ma <[email protected]>
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
Fix three problems observed while running the tests in the CI recently:

1. Static check

Fix spelling issue captured by static check (astronomer#1000)

2. Integration test that's no longer needed

The issue we were trying to capture for no longer happens in the latest
version of the Apache Airflow provider
`apache-airflow-providers-postgres==5.11.1rc1`:
apache/airflow#39842

3. Skip a buggy version of OL

There was a breaking change between
`openlineage-integration-common==1.14.0` and
`openlineage-integration-common==1.15.0` . It may have been an
unintended side-effect of
OpenLineage/OpenLineage#2693.

This is an example of how Cosmos was using `DbtLocalArtifactProcessor` -
something that had been agreed upon in the past:

```
      openlineage_processor = DbtLocalArtifactProcessor(
            producer=OPENLINEAGE_PRODUCER,
            job_namespace=LINEAGE_NAMESPACE,
            project_dir=project_dir,
            profile_name=self.profile_config.profile_name,
            target=self.profile_config.target_name,
        )
        events = openlineage_processor.parse()
        for completed in events.completes:
            for output in getattr(completed, source):
                dataset_uri = output.namespace + "/" + output.name
                uris.append(dataset_uri)
```

In `openlineage-integration-common==1.14.0` and earlier versions, this
would create URIs in the format:
```
postgres:https://0.0.0.0:5432/postgres.public.stg_customers
```
Since openlineage-integration-common==1.15.0 , this leads to URIs being
created in the format:
```
postgres.public.stg_customers/postgres:https://0.0.0.0:5432
```

This was fixed in OpenLineage/OpenLineage#2735
and released as part of OL 1.16:
https://github.com/OpenLineage/OpenLineage/releases/tag/1.16.0
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
Bug fixes

* Fix the invocation mode for ``ExecutionMode.VIRTUALENV`` by @marco9663
in astronomer#1023
* Fix Cosmos ``enable_cache`` setting by @tatiana in astronomer#1025
* Make ``GoogleCloudServiceAccountDictProfileMapping`` dataset profile
arg optional by @oliverrmaa and @pankajastro in astronomer#839 and astronomer#1017
* Athena profile mapping set ``aws_session_token`` in profile only if it
exists by @pankajastro in astronomer#1022

Others

* Update dbt and Airflow conflicts matrix by @tatiana in astronomer#1026
* Enable Python 3.12 unittest by @pankajastro in astronomer#1018
* Improve error logging in ``DbtLocalBaseOperator`` by @davidsteinar in
astronomer#1004
* Add GitHub issue templates for bug reports and feature request by
@pankajkoti in astronomer#1009
* Add more fields in bug template to reduce turnaround in issue triaging
by @pankajkoti in astronomer#1027
* Fix ``dev/Dockerfile`` + Add ``uv pip install`` for faster build time
by @dwreeves in astronomer#997
* Drop support for Airflow 2.3 by @pankajkoti in astronomer#994
* Update Astro Runtime image by @RNHTTR in astronomer#988 and astronomer#989
* Enable ruff F linting by @pankajastro in astronomer#985
* Move Cosmos Airflow configuration to settings.py by @pankajastro in
astronomer#975
* Fix CI Issues by @tatiana in astronomer#1005
* Pre-commit hook updates in astronomer#1000, astronomer#1019

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Reeves <[email protected]>
Co-authored-by: Pankaj Koti <[email protected]>
Co-authored-by: davidsteinar <[email protected]>
Co-authored-by: David Steinar Asgrimsson <[email protected]>
Co-authored-by: Marco Yuen <[email protected]>
Co-authored-by: Pankaj Singh <[email protected]>
Co-authored-by: Ollie Ma <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:ci Related to CI, Github Actions, or other continuous integration tools area:dependencies Related to dependencies, like Python packages, library versions, etc lgtm This PR has been approved by a maintainer profile:postgres Related to Postgres ProfileConfig size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants