-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: BigQueryIO direct read not reading all rows when set --setEnableBundling=true #26354
Comments
Affecting >=v2.46.0 demo that reproduces this issue: https://github.com/Abacn/beam-demo/tree/bigqueryiotestbranch |
This seems like a pretty severe issue. Any progress? |
The feature is introduced in Beam v2.46.0 and is activated only when this currently undocumented pipeline option is set. No production user is using it. @vachan-shetty is working on fix. Feel free to assign to yourself. |
Is this related to #26521 or no? |
@kennknowles thanks, will test if #26503 fixed the issue. Update, no, that issue is for write, this is for read |
What happened?
The feature is introduced in #25392. When
--setEnableBundling=true
pipeline option is set, it turns out that BigQueryIO only reads a small fraction of row for large table. Reproduced readingtpcds_1T.web_sales
table.Number of rows: 720,000,376
--setEnableBundling=true
: 44,550,489 rows read--setEnableBundling=false
: 720,000,376 rows readReading from
tpcds_1G.web_sales
table, the issue is not triggered, as 18,000 rows are read.Issue Priority
Priority: 1 (data loss / total loss of function)
Issue Components
The text was updated successfully, but these errors were encountered: