Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make FileOutputCommiter as default #86

Merged
merged 3 commits into from
Aug 25, 2018
Merged

Make FileOutputCommiter as default #86

merged 3 commits into from
Aug 25, 2018

Conversation

ajayborra
Copy link
Contributor

@ajayborra ajayborra commented Aug 24, 2018 •
edited
Loading

Related issues
#54

Changes

  • Made FileOutputCommiter default.
  • Updated docs.
  • Removed DOC custom classes.

@ajayborra ajayborra mentioned this pull request Aug 24, 2018
Closed
@codecov
Copy link

codecov bot commented Aug 24, 2018 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (master@ed98173). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master      #86   +/-   ##
=========================================
  Coverage          ?   86.07%           
=========================================
  Files             ?      292           
  Lines             ?     9521           
  Branches          ?      548           
=========================================
  Hits              ?     8195           
  Misses            ?     1326           
  Partials          ?        0
Impacted Files Coverage Δ
...la/com/salesforce/op/utils/io/avro/AvroInOut.scala 100% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed98173...0963802. Read the comment docs.

tovbinm
tovbinm reviewed Aug 24, 2018
View reviewed changes
docs/installation/index.md Outdated
@@ -5,3 +5,13 @@
* Clone the TransmogrifAI repo: `git clone https://github.com/salesforce/TransmogrifAI.git`
* Build the project: `cd TransmogrifAI && ./gradlew compileTestScala installDist`
* Start hacking

# (Optional) Configuration
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not aure if this belongs here or we need a new section. @snabar @leahmcguire wdyt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need to document output committer conf prominently since this is well documented for Hadoop and by extension for Spark itself. So probably not the time to modify index.md at this time either.

tovbinm
tovbinm reviewed Aug 24, 2018
View reviewed changes
Copy link
Collaborator

@tovbinm tovbinm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please grep the whole codebase for the removed class names and make aure you remove them all. I think build.gradle has some reference to them in license plugin config for instance.

@ajayborra
Copy link
Contributor Author

@tovbinm ran a case-insensitive grep for both classes. Didn't find any new occurrences.

gerashegalov
gerashegalov suggested changes Aug 24, 2018
View reviewed changes
Copy link
Contributor

@gerashegalov gerashegalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but let us not duplicate hadoop/spark documentation unless on a wiki or FAQ if it comes up a lot.

@tovbinm
Copy link
Collaborator

tovbinm commented Aug 24, 2018

@ajayborra have you tried running the helloworld after the change? Otherwise lgtm

tovbinm
tovbinm approved these changes Aug 24, 2018
View reviewed changes
Copy link
Collaborator

@tovbinm tovbinm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please test with helloword, otherwise lgtm

@ajayborra
Copy link
Contributor Author

@tovbinm Yes ran iris dataset example by setting output destination to s3. Status is green.

@tovbinm tovbinm merged commit 7eb2a8d into salesforce:master Aug 25, 2018
@ajayborra ajayborra deleted the ab/use_default_committers branch August 25, 2018 14:51
ericwayman pushed a commit that referenced this pull request Feb 8, 2019
@ajayborra @tovbinm
Make FileOutputCommiter as default (#86)
e515e32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tovbinm tovbinm tovbinm approved these changes

@gerashegalov gerashegalov gerashegalov approved these changes

@leahmcguire leahmcguire Awaiting requested review from leahmcguire leahmcguire is a code owner

Assignees
No one assigned
Labels
cla:signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ajayborra @tovbinm @gerashegalov