Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-701] Common API based on SAM interfaces rather than rich functions #85

Closed
wants to merge 14 commits into from
Closed

[FLINK-701] Common API based on SAM interfaces rather than rich functions #85

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
9bdc931
Renamed of Function to RichFunction, created empty Function interface
ktzoumas Jul 18, 2014
d8bdd90
Renamed of Function to RichFunction, created empty Function interface
ktzoumas Jul 18, 2014
56b27cc
changes to wrapping function
ktzoumas Jul 18, 2014
2eb3aa8
map and reduce operators
ktzoumas Jul 19, 2014
762c33b
group reduce refactoring
ktzoumas Jul 23, 2014
d041d15
group reduce as interface -- shallow implementation
ktzoumas Jul 24, 2014
aed2549
Java and record API based on SAM interfaces
ktzoumas Jul 29, 2014
b8b5c6f
Propose test structure, DeltaIterationTraslationTest passing, renamed…
ktzoumas Jul 30, 2014
5c9c149
delete accidentally added file
ktzoumas Jul 30, 2014
94687a6
added missing licenses
ktzoumas Jul 30, 2014
ba38b90
removed dummy emacs cache pom file
ktzoumas Jul 30, 2014
2e1d2b6
Fixed java8-tests pom file, lambda detection using SerializedLambda
ktzoumas Jul 31, 2014
dcb54d0
Renaming of SAMs to *Function and rich functions to Rich*Function, ch…
ktzoumas Jul 31, 2014
4868dd9
Renaming of SAMs to *Function and rich functions to Rich*Function, ch…
ktzoumas Jul 31, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Propose test structure, DeltaIterationTraslationTest passing, renamed… 
… GeneratedJoinFunction
  • Loading branch information
@ktzoumas
ktzoumas committed Jul 30, 2014
commit b8b5c6f9900740203961bf10a9508f2f50bd4a81
650 changes: 650 additions & 0 deletions #pom.xml#
View file
Open in desktop

Large diffs are not rendered by default.

11 changes: 0 additions & 11 deletions flink-clients/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,17 +154,6 @@ under the License.
</execution>
</executions>
</plugin>
<plugin>
<!-- just define the Java version to be used for compiling and plugins -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version><!--$NO-MVN-MAN-VER$-->
<configuration>
<source>1.8</source>
<target>1.8</target>
<!-- High optimization, no debugging <compilerArgument>-g:none -O</compilerArgument> -->
</configuration>
</plugin>
</plugins>

<pluginManagement>
Expand Down
13 changes: 0 additions & 13 deletions flink-examples/flink-java-examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,19 +317,6 @@ under the License.

</executions>
</plugin>

<plugin>
<!-- just define the Java version to be used for compiling and plugins -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version><!--$NO-MVN-MAN-VER$-->
<configuration>
<source>1.8</source>
<target>1.8</target>
<!-- High optimization, no debugging <compilerArgument>-g:none -O</compilerArgument> -->
</configuration>
</plugin>

</plugins>
</build>

Expand Down
17 changes: 0 additions & 17 deletions flink-examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,21 +50,4 @@ under the License.
<module>flink-java-examples</module>
<module>flink-scala-examples</module>
</modules>

<build>
<plugins>
<plugin>
<!-- just define the Java version to be used for compiling and plugins -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version><!--$NO-MVN-MAN-VER$-->
<configuration>
<source>1.8</source>
<target>1.8</target>
<!-- High optimization, no debugging <compilerArgument>-g:none -O</compilerArgument> -->
</configuration>
</plugin>
</plugins>
</build>

</project>
16 changes: 0 additions & 16 deletions flink-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,22 +34,6 @@ under the License.

<packaging>jar</packaging>

<build>
<plugins>
<plugin>
<!-- just define the Java version to be used for compiling and plugins -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version><!--$NO-MVN-MAN-VER$-->
<configuration>
<source>1.6</source>
<target>1.6</target>
<!-- High optimization, no debugging <compilerArgument>-g:none -O</compilerArgument> -->
</configuration>
</plugin>
</plugins>
</build>

<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
Expand Down
6 changes: 4 additions & 2 deletions ...c/main/java/org/apache/flink/api/java/functions/UnsupportedLambdaExpressionException.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@

package org.apache.flink.api.java.functions;

public class UnsupportedLambdaExpressionException extends RuntimeException {
import org.apache.flink.api.common.InvalidProgramException;

public class UnsupportedLambdaExpressionException extends InvalidProgramException {

private static final long serialVersionUID = -1721898801986321010L;

public UnsupportedLambdaExpressionException() {
super("Java 8 lambda expressions are currently only supported in filter and reduce user-defined functions.");
super("Java 8 lambda expressions are currently supported only in filter and reduce user-defined functions.");
}
}
10 changes: 3 additions & 7 deletions flink-java/src/main/java/org/apache/flink/api/java/operators/JoinOperator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,27 +481,23 @@ public <R> EquiJoin<I1, I2, R> with (Joinable<I1, I2, R> function) {
if (FunctionUtils.isLambdaFunction(function)) {
throw new UnsupportedLambdaExpressionException();
}
FlatJoinable generatedFunction = new GeneratedFlatJoinFunction<I1, I2, R> (function);
FlatJoinable generatedFunction = new WrappingFlatJoinFunction<I1, I2, R>(function);
TypeInformation<R> returnType = TypeExtractor.getJoinReturnTypes(function, getInput1Type(), getInput2Type());
return new EquiJoin<I1, I2, R>(getInput1(), getInput2(), getKeys1(), getKeys2(), generatedFunction, function, returnType, getJoinHint());
}

private static class GeneratedFlatJoinFunction<IN1, IN2, OUT> extends WrappingFunction<Joinable<IN1,IN2,OUT>> implements FlatJoinable<IN1, IN2, OUT> {
private static class WrappingFlatJoinFunction<IN1, IN2, OUT> extends WrappingFunction<Joinable<IN1,IN2,OUT>> implements FlatJoinable<IN1, IN2, OUT> {

private static final long serialVersionUID = 1L;

private GeneratedFlatJoinFunction(Joinable<IN1, IN2, OUT> wrappedFunction) {
private WrappingFlatJoinFunction(Joinable<IN1, IN2, OUT> wrappedFunction) {
super(wrappedFunction);
}

@Override
public void join(IN1 left, IN2 right, Collector<OUT> out) throws Exception {
out.collect (this.wrappedFunction.join(left, right));
}

public Joinable getWrappedFunction () {
return this.wrappedFunction;
}
}

/*
Expand Down
4 changes: 3 additions & 1 deletion ...-java/src/main/java/org/apache/flink/api/java/operators/translation/WrappingFunction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,9 @@ public void setRuntimeContext(RuntimeContext t) {
}
}


public T getWrappedFunction () {
return this.wrappedFunction;
}


private static class WrappingRuntimeContext implements RuntimeContext {
Expand Down
10 changes: 8 additions & 2 deletions ...t/java/org/apache/flink/api/java/operators/translation/DeltaIterationTranslationTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,14 @@ public void testCorrectTranslation() {

assertEquals(IdentityMapper.class, worksetMapper.getUserCodeWrapper().getUserCodeClass());
assertEquals(NextWorksetMapper.class, nextWorksetMapper.getUserCodeWrapper().getUserCodeClass());
//assertEquals(SolutionWorksetJoin.class, solutionSetJoin.getUserCodeWrapper().getUserCodeClass());

if (solutionSetJoin.getUserCodeWrapper().getUserCodeObject() instanceof WrappingFunction) {
WrappingFunction wf = (WrappingFunction) solutionSetJoin.getUserCodeWrapper().getUserCodeObject();
assertEquals(SolutionWorksetJoin.class, wf.getWrappedFunction().getClass());
}
else {
assertEquals(SolutionWorksetJoin.class, solutionSetJoin.getUserCodeWrapper().getUserCodeClass());
}

assertEquals(BEFORE_NEXT_WORKSET_MAP, nextWorksetMapper.getName());

assertEquals(AGGREGATOR_NAME, iteration.getAggregators().getAllRegisteredAggregators().iterator().next().getName());
Expand Down
196 changes: 196 additions & 0 deletions flink-java8-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http:https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

-->
<project xmlns="http:https://maven.apache.org/POM/4.0.0" xmlns:xsi="http:https://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http:https://maven.apache.org/POM/4.0.0 http:https://maven.apache.org/maven-v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.apache.flink</groupId>
<artifactId>flink-parent</artifactId>
<version>0.6-incubating-SNAPSHOT</version>
<relativePath>..</relativePath>
</parent>

<artifactId>flink-java8-tests</artifactId>
<name>flink-java8-tests</name>

<packaging>jar</packaging>

<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need all dependencies here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for catching this, this is now fixed

<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-compiler</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.7</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java-examples</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala-examples</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-tests</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<!-- just define the Java version to be used for compiling and plugins -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The maven-compiler-plugin is defined twice in the file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for catching this, this is now fixed

<version>3.1</version><!--$NO-MVN-MAN-VER$-->
<configuration>
<source>1.8</source>
<target>1.8</target>
<!-- High optimization, no debugging <compilerArgument>-g:none -O</compilerArgument> -->
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<log.level>WARN</log.level>
</systemPropertyVariables>
<forkMode>once</forkMode>
<argLine>-Xmx1024m</argLine>
</configuration>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<log.level>WARN</log.level>
</systemPropertyVariables>
<forkMode>always</forkMode>
<threadCount>1</threadCount>
<perCoreThreadCount>false</perCoreThreadCount>
</configuration>
</plugin>
<plugin>
<!-- Define the Java version to be used for compiling and plugins -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version><!--$NO-MVN-MAN-VER$-->
<configuration>
<source>1.8</source>
<target>1.8</target>
<!-- High optimization, no debugging <compilerArgument>-g:none -O</compilerArgument> -->
</configuration>
</plugin>

</plugins>

<pluginManagement>
<plugins>
<!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
<plugin>
<groupId>org.eclipse.m2e</groupId>
<artifactId>lifecycle-mapping</artifactId>
<version>1.0.0</version>
<configuration>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>
org.apache.maven.plugins
</groupId>
<artifactId>
maven-assembly-plugin
</artifactId>
<versionRange>
[2.4,)
</versionRange>
<goals>
<goal>single</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore></ignore>
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
Loading