Skip to content

Instantly share code, notes, and snippets.

@Atrix1987

Atrix1987/PipelineSnippet.java

Last active February 9, 2017 12:25
Show Gist options
  • Save Atrix1987/402843d494ae387e09bf1d1d706a7fb3 to your computer and use it in GitHub Desktop.
Save Atrix1987/402843d494ae387e09bf1d1d706a7fb3 to your computer and use it in GitHub Desktop.
Download ZIP
Snippet of an amplitude data import pipeline
Raw
PipelineSnippet.java
//Configuring the options etc
Pipeline p = Pipeline.create(options);
p.apply(TextIO.Read.named("ReadFiles").from(options.getInputFile()).withCompressionType(CompressionType.GZIP))
.apply(new ProcessRecords(eventType))
.apply(BigQueryIO.Write.to(table.getTableReference()).withSchema(table.getSchema())
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND));
p.run();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment