Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/1331 disables deduplication for incremental #1892

Open
wants to merge 9 commits into
base: devel
Choose a base branch
from

Conversation

willi-mueller
Copy link
Collaborator

@willi-mueller willi-mueller commented Sep 28, 2024

Description

This PR disables deduplication for the test case described by ingestr here: #971 (comment)

Related Issues

Questions

@rudolfix

  1. I could not understand your points 2, 3, and 4 in the issue disable rows deduplication if Incremental is attached to a resource with merge write disposition #1131 . Are they already implemented in this PR?
  2. I am not sure if the last test tests/extract/test_incremental.py::test_deduplication_on_write_disposition_not_merge makes sense at all because on write_disposition="replace" the table is truncated before the load. I included it because the ticket speaks of the merge write disposition so I wanted to test the opposite too. Feel free to drop this commit.
  3. Are the assertions on the incremental's last_value superfluous?
  4. I could not find a way to test that the incremental returns all values without deduplication. Thus, I implemented the tests by making assertions on the loaded data. Is this strategy fine?

TODO after merge

  • notify the developers of ingestr

Copy link

netlify bot commented Sep 28, 2024

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 4c3bd87
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/6730c0363e24740008651ae8

@willi-mueller willi-mueller force-pushed the feat/1331-disable-deduplication-for-merge branch from 288697d to d4b8b12 Compare September 28, 2024 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

disable rows deduplication if Incremental is attached to a resource with merge write disposition
2 participants