Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mypy + some other misc. changes #54

Merged
merged 5 commits into from
Aug 23, 2023
Merged

Conversation

dwreeves
Copy link
Contributor

@dwreeves dwreeves commented Aug 18, 2023

Resolves #53

Changes

Docstring clarification for FivetranSensor

I added this because I thought I was working on #49 when I first started coding.

In any event, I added language clarifying how FivetranSensor works more specifically than what was there.

Type-checking

A lot of hooks.py was un-annotated, so I made sure to get the whole thing annotated.

I then added Mypy to the pre-commit. I also added mypy to the CI.

_do_api_call() and _do_api_call_async() API changes

I made these more plainly wrap the requests.request() and asyncio.ClientSession.request APIs. It did not seem like there was much reason for these APIs to deviate. Deviations include:

  • It had a parameter called json that maps to either the data=json or params=json kwargs in the requests.request() call. This seems less desirable than just passing data=data, params=params, or json=json. Specifically, the json kwarg is confusing from the perspective of the requests and aiohttp APIs: in those APIs, json= expects a dict and data= expects a string, but in the FivetranHook API, json= is a string that gets mapped to data=.
  • The first 2 args were a tuple for method and URL, whereas these could just be args (like requests.request())
  • For the internals: it is unnecessarily burdensome from a Mypy perspective, and also inflexible from a future-proofing perspective, to do an if-elif on the method to get the implementation when requests.request() can do that for you.

That said, changing this is probably annoying for users as there may be many users using this function. So I made sure the old API works and gave a deprecation warning via the following:

if is_container(method) and len(method) == 2:
    import warnings
    warnings.warn(
        "The API for _do_api_call() has changed to closer match the"
        " requests.request() API. Please update your code accordingly.",
        DeprecationWarning,
        stacklevel=2
    )
    method, endpoint = method
elif endpoint is None:
    raise TypeError(
        f"{self.__class__.__name__}._do_api_call() missing 1 required"
        f" positional argument: 'endpoint'"
    )

Lastly, I removed the test for "UNKNOWN". The reason why is because requests.request() can handle any method passed in, and the Fivetran API itself should be in charge of rejecting bad methods. (Note that this is the only nontrivial test change I made; all other test cases pass with essentially no changes, meaning my code is more or less a refactor.)

TLDR:

  • _do_api_call() now matches requests.request()
  • I made this backwards compatible with a DeprecationWarning

"Breaking" changes

I put "breaking" in quotes because these are very minor API features that were not being used by anything. That said they still constitute breaking changes, in some sense, if we consider changing the FivetranHook's method return types to constitute breaking changes (even if those return types are not so useful).

  • FivetranHook.check_connector now returns the connector_details dict object. (So that prep_connector can avoid making an unnecessary 3rd API call.)
  • FivetranHook.prep_connector now returns None instead of Literal[True].

Use hook as cached property

It is more idiomatic in Airflow code for the hook for an operator to be called as a cached_property, so I swapped the operator and sensor code to implement this pattern, and added a deprecation warning for _get_hook().

Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look like ci is not happy

fivetran_provider_async/hooks.py Outdated Show resolved Hide resolved
fivetran_provider_async/hooks.py Outdated Show resolved Hide resolved
fivetran_provider_async/operators.py Outdated Show resolved Hide resolved
@dwreeves
Copy link
Contributor Author

CI failed because isort moved the type: ignore[import] comment... 😅 Fixed!

Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @dwreeves for the contribution.

@pankajastro pankajastro merged commit cd1a335 into astronomer:main Aug 23, 2023
6 checks passed
@dwreeves
Copy link
Contributor Author

No problem, thanks for the merge. I will work on #49 next.

@dwreeves dwreeves deleted the add-mypy branch November 30, 2023 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make code more idiomatic, clean, and better enforced
2 participants