Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-8949] SpannerIO integration tests #11210

Merged
merged 16 commits into from
May 14, 2020
Merged

[BEAM-8949] SpannerIO integration tests #11210

merged 16 commits into from
May 14, 2020

Conversation

mszb
Copy link
Contributor

@mszb mszb commented Mar 24, 2020 •
edited
Loading

Added Spanner io integration test.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status
Build Status		 Build Status
Build Status
Build Status		 Build Status Build Status Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status		 --- Build Status
Build Status
Build Status		 Build Status
Build Status		 --- --- Build Status
XLang --- --- --- Build Status --- --- Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status		 Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@aaltay
Copy link
Member

aaltay commented Mar 24, 2020

retest this please

@iemejia
Copy link
Member

iemejia commented Mar 25, 2020

retest this please

@mszb
Copy link
Contributor Author

mszb commented Mar 27, 2020

@iemejia @aaltay. Could you please run the post-commit tests as well!

Run Python PostCommit

Thanks!

@iemejia
Copy link
Member

iemejia commented Mar 27, 2020

Run Python PostCommit

@mszb mszb changed the title [WIP] [BEAM-8949] SpannerIO Read integration tests [WIP] [BEAM-8949] SpannerIO integration tests Apr 13, 2020
@mszb mszb changed the title [WIP] [BEAM-8949] SpannerIO integration tests [BEAM-8949] SpannerIO integration tests Apr 13, 2020
@aaltay
Copy link
Member

aaltay commented Apr 13, 2020

retest this please

@aaltay
Copy link
Member

aaltay commented Apr 13, 2020

Run Python PostCommit

@aaltay
Copy link
Member

aaltay commented Apr 13, 2020

Run Python 3.5 PostCommit

@iemejia
Copy link
Member

iemejia commented Apr 20, 2020

retest this please

@lukecwik
Copy link
Member

Run Python PostCommit

@lukecwik
Copy link
Member

Run Python 3.5 PostCommit

@iemejia
Copy link
Member

iemejia commented Apr 20, 2020

retest this please

2 similar comments
@iemejia
Copy link
Member

iemejia commented Apr 20, 2020

retest this please

@iemejia
Copy link
Member

iemejia commented Apr 20, 2020

retest this please

@iemejia
Copy link
Member

iemejia commented Apr 20, 2020

mmmmmm

@aaltay
Copy link
Member

aaltay commented May 5, 2020

@mszb - Do you know why the test is failing?

@mszb
Copy link
Contributor Author

mszb commented May 5, 2020

@aaltay - Wordcount stream tests are failing. They are running successfully on my local machine. Jira ticket is already created for that - https://issues.apache.org/jira/browse/BEAM-9767
Maybe retriggering the pre-commit test works but can't say that would pass 100% bcoz of the flaky wordcount test.

Apart from that, there were 3 pre-commit jobs are triggered almost the same time on Jenkins (with ~30seconds difference) and one of them is successfully completed but sadly the failed one is linked with the github.

successful pre-commit: https://builds.apache.org/job/beam_PreCommit_Python_Commit/12541/

failed jobs:
https://builds.apache.org/job/beam_PreCommit_Python_Commit/12542/
https://builds.apache.org/job/beam_PreCommit_Python_Commit/12543/

@aaltay
Copy link
Member

aaltay commented May 5, 2020

Run Python PreCommit

@aaltay
Copy link
Member

aaltay commented May 5, 2020 •
edited
Loading

@aaltay - Wordcount stream tests are failing. They are running successfully on my local machine. Jira ticket is already created for that - https://issues.apache.org/jira/browse/BEAM-9767
Maybe retriggering the pre-commit test works but can't say that would pass 100% bcoz of the flaky wordcount test.

Apart from that, there were 3 pre-commit jobs are triggered almost the same time on Jenkins (with ~30seconds difference) and one of them is successfully completed but sadly the failed one is linked with the github.

successful pre-commit: https://builds.apache.org/job/beam_PreCommit_Python_Commit/12541/

failed jobs:
https://builds.apache.org/job/beam_PreCommit_Python_Commit/12542/
https://builds.apache.org/job/beam_PreCommit_Python_Commit/12543/

OK, let's re-run to get a clear signal.

chamikaramj
chamikaramj reviewed May 8, 2020
View reviewed changes
Copy link
Contributor

@chamikaramj chamikaramj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

sdks/python/apache_beam/io/gcp/experimental/spannerio_test.py Outdated Show resolved Hide resolved
sdks/python/apache_beam/io/gcp/experimental/spannerio.py Outdated Show resolved Hide resolved
sdks/python/apache_beam/io/gcp/experimental/spannerio.py Outdated Show resolved Hide resolved
@mszb
Copy link
Contributor Author

mszb commented May 10, 2020

@chamikaramj: i've made some changes, could you please trigger the tests!
-Thanks

@mszb mszb requested a review from chamikaramj May 10, 2020 17:54
@chamikaramj
Copy link
Contributor

Thanks. LGTM.

@chamikaramj
Copy link
Contributor

Retest this please

1 similar comment
@chamikaramj
Copy link
Contributor

Retest this please

@chamikaramj
Copy link
Contributor

Run Python PostCommit

3 similar comments
@chamikaramj
Copy link
Contributor

Run Python PostCommit

@chamikaramj
Copy link
Contributor

Run Python PostCommit

@chamikaramj
Copy link
Contributor

Run Python PostCommit

@chamikaramj
Copy link
Contributor

Run Python 3.7 PostCommit

2 similar comments
@udim
Copy link
Member

udim commented May 12, 2020

Run Python 3.7 PostCommit

@aaltay
Copy link
Member

aaltay commented May 13, 2020

Run Python 3.7 PostCommit

@mszb
Copy link
Contributor Author

mszb commented May 13, 2020

There were three post-commit jobs triggered for this PR, and the first one is passed.

https://builds.apache.org/job/beam_PostCommit_Python37_PR/144/ (PASSED)
https://builds.apache.org/job/beam_PostCommit_Python37_PR/145/ (Bigquery test failed)
https://builds.apache.org/job/beam_PostCommit_Python37_PR/146/ (Bigquery test failed)

should we try to re-run the job?

@pabloem
Copy link
Member

pabloem commented May 13, 2020

it seems that BQ tests were broken. Can you rebase the PR, and we'll rerun the tests?

@mszb
Copy link
Contributor Author

mszb commented May 14, 2020

it seems that BQ tests were broken. Can you rebase the PR, and we'll rerun the tests?

done!

@aaltay
Copy link
Member

aaltay commented May 14, 2020

retest this please

@pabloem
Copy link
Member

pabloem commented May 14, 2020

retest this plesae

@pabloem
Copy link
Member

pabloem commented May 14, 2020

Run Python 3.7 PostCommit

1 similar comment
@pabloem
Copy link
Member

pabloem commented May 14, 2020

Run Python 3.7 PostCommit

@pabloem pabloem merged commit aba1db8 into apache:master May 14, 2020
yirutang pushed a commit to yirutang/beam that referenced this pull request Jul 23, 2020
@yirutang
Merge pull request apache#11210 from [BEAM-8949] SpannerIO integratio…
a64457a 
…n tests

* wip: added spannerio read integration tests

* adds spanner io write it tests

* fix formatter

* fix batch pipeline and it test

* fix test db name

* fix linting issues

* fix failed batch test cases and updates changes.md

* revert changes from spanner io python connector and update IT test validation code.

* adds spanner it tests in the py common gradle build file

Co-authored-by: Shoaib <[email protected]>
Co-authored-by: Shoaib <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pabloem pabloem Awaiting requested review from pabloem

@chamikaramj chamikaramj Awaiting requested review from chamikaramj

Assignees
No one assigned
Labels
build gcp io python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@mszb @aaltay @iemejia @lukecwik @pabloem @chamikaramj @udim