Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-9886] Auto-inferring project for ReadFromBigQuery #11560

Merged
merged 11 commits into from
May 5, 2020

Conversation

pabloem
Copy link
Member

@pabloem pabloem commented Apr 28, 2020

r: @kamilwu @y1chi

This helps infer the project where we should execute export jobs / queries / create temp tables, etc


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
--- Build Status
Build Status
Build Status
Build Status
Build Status
--- --- Build Status
XLang --- --- --- Build Status --- --- Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status
Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@pabloem
Copy link
Member Author

pabloem commented Apr 28, 2020

Run Portable_Python PreCommit

@pabloem
Copy link
Member Author

pabloem commented Apr 29, 2020

Run Python 3.7 PostCommit

@pabloem
Copy link
Member Author

pabloem commented Apr 29, 2020

Run Python 3.7 PostCommit

@pabloem
Copy link
Member Author

pabloem commented Apr 29, 2020

@pabloem
Copy link
Member Author

pabloem commented Apr 29, 2020

Run Python PreCommit

@pabloem
Copy link
Member Author

pabloem commented Apr 29, 2020

Run Python 3.7 PostCommit

@pabloem
Copy link
Member Author

pabloem commented May 1, 2020

Run Python 3.7 PostCommit

@pabloem
Copy link
Member Author

pabloem commented May 2, 2020

Run Python2_PVR_Flink PreCommit

@pabloem
Copy link
Member Author

pabloem commented May 2, 2020

@pabloem
Copy link
Member Author

pabloem commented May 2, 2020

Run Python Precommit

@pabloem
Copy link
Member Author

pabloem commented May 2, 2020

Run Python2_PVR_Flink PreCommit

1 similar comment
@pabloem
Copy link
Member Author

pabloem commented May 2, 2020

Run Python2_PVR_Flink PreCommit

@pabloem
Copy link
Member Author

pabloem commented May 2, 2020

Python2_PVR_Flink tests are broken on master : /

@pabloem
Copy link
Member Author

pabloem commented May 4, 2020

Run Python2_PVR_Flink PreCommit

@pabloem
Copy link
Member Author

pabloem commented May 4, 2020

@kamilwu @y1chi PTAL

@@ -526,7 +526,7 @@ def reader(self, test_bigquery_client=None):


def _to_bool(value):
return value == 'true'
return value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you explain why this change is needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the coder incorrectly expected boolean types to be encoded as strings. This is incorrect, as JSON supports boolean 'natively', like this: https://json-schema.org/understanding-json-schema/reference/boolean.html

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the _to_bool function then, would the bool casting suffice?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhh what a great catch. thanks Yichi!

Copy link
Contributor

@y1chi y1chi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks Pablo.

@pabloem
Copy link
Member Author

pabloem commented May 5, 2020

Run Python 3.7 PostCommit

@pabloem pabloem changed the title Auto-inferring project for ReadFromBigQuery [BEAM-9886] Auto-inferring project for ReadFromBigQuery May 5, 2020
@pabloem pabloem merged commit 34c58c4 into apache:master May 5, 2020
@pabloem pabloem deleted the autoproject branch May 5, 2020 02:06
yirutang pushed a commit to yirutang/beam that referenced this pull request Jul 23, 2020
…ct for ReadFromBigQuery

* Auto-inferring project for ReadFromBigQuery

* Fixup

* fixup

* fixup

* Formatter fixup

* Fixup

* fixformat

* Fix coder issue

* fixup

* Fix lint

* Fixup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants