Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BEAM-2233,BEAM-2223] Adjust TestSparkPipelineOptions registration; add per-runner profiles to Java 8 examples pom #3002

Closed

Conversation

kennknowles
Copy link
Member

@kennknowles kennknowles commented May 9, 2017

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

  • Make sure the PR title is formatted like:
    [BEAM-<Jira issue #>] Description of pull request
  • Make sure tests pass via mvn clean verify.
  • Replace <Jira issue #> in the title with the actual Jira issue
    number, if there is one.
  • If this contribution is large, please file an Apache
    Individual Contributor License Agreement.

We have cascading failures:

  • All runners were included together in the include-runners profile in the Java 8 examples
  • TestOptions has a hard dep on hamcrest that we list as provided, putting a burden on users
  • TestSparkPipelineOptions was registered, so the Spark runner had this same hard dep and class load time
  • If you do options.as(MyUnregisteredOptions.class) this will cause TestPipeline/PipelineOptionsFactory to crash with a parse error even if none of the unregistered options were present

These should all be fixed, IMO as follows:

  • Runners should have separate profiles (done in this PR)
  • TestOptions should either not use hamcrest or should declare a correct dependency (not resolved)
  • TestSparkPipelineOptions should either not need to be registered or should be allowed to be registered (expected resolution: no need to register)
  • TestPipeline / PipelineOptionsFactory should not crash due to how options have been observed, when they are not mutated, and probably should just not crash at all

@kennknowles kennknowles force-pushed the java8-examples-runner-profiles branch from b45ab8b to 90dc726 Compare May 9, 2017 21:25
@kennknowles kennknowles changed the title [BEAM-2233,BEAM-2223] Add per-runner profiles to Java 8 examples pom [BEAM-2233,BEAM-2223] Adjust TestSparkPipelineOptions registration; add per-runner profiles to Java 8 examples pom May 9, 2017
@kennknowles
Copy link
Member Author

run spark validatesrunner

@kennknowles
Copy link
Member Author

retest this please

@kennknowles
Copy link
Member Author

I have manually run the Spark ValidatesRunner tests and eventually got them to succeed via the method of registering the test options only in src/test.

Copy link
Member

@davorbonaci davorbonaci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kennknowles kennknowles force-pushed the java8-examples-runner-profiles branch from 90dc726 to 35ec567 Compare May 9, 2017 22:10
@kennknowles
Copy link
Member Author

run spark validatesrunner

@davorbonaci
Copy link
Member

All passed. Merging.

@asfgit asfgit closed this in b9ea415 May 10, 2017
@kennknowles kennknowles deleted the java8-examples-runner-profiles branch May 26, 2017 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants