[BEAM-9951] Adding integration tests for synthetic pipelines in Go #11870
Conversation
Still in progress. This adds integration tests for synthetic pipelines.
|
As a note for what still needs to be done:
CC: @lostluck Take a look if you want, but don't worry about reviewing yet. |
| col = beam.DropKey(s, col) | ||
| } | ||
| counted := beam.Combine(s, &elmCountCombineFn{}, col) | ||
| Equals(s, counted, count) |
There was a problem hiding this comment.
Can the Sum transform be re-used instead?
There was a problem hiding this comment.
I think it could, and I considered it, but the error message isn't too helpful either (better than equals though). I added a ParDo0 with a custom error message which I think ends up being the most user-friendly.
There was a problem hiding this comment.
I meant passert.Sum. Sorry that should have been clearer.
There was a problem hiding this comment.
I thought so, that's what I was referring to. The error message in passert.Sum (while better than passert.Equals) is still a bit unclear, just because it specifically mentions passert.Sum instead of passert.Count.
Adding testing, and adjusting the error message for the count transform.
youngoli
changed the title
|
I added unit tests and changed the error message. I spent a bunch of time moving the Count code to Beam (specifically the stats package), but ended up having to scrap it because having it there caused a circular dependency. Testing relied on stats which relied on testing for passerts. I'll add an independent version to the stats package because I already have the code written and it seems valuable, but it's not going to be used for the passerts. |
youngoli
force-pushed
the
sdf-implementation
branch
from
9fbfdd1
to
c79fdef
Compare
|
Run Go PostCommit |
This adds integration tests for synthetic pipelines.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.