[FLINK-15672][build] Migrate to log4j2

mbalassi · Feb 21, 2020 · e396e7c · e396e7c
1 parent a50b666
commit e396e7c
Show file tree

Hide file tree

Showing 62 changed files with 720 additions and 607 deletions.
diff --git a/docs/monitoring/logging.md b/docs/monitoring/logging.md
@@ -23,21 +23,40 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-The logging in Flink is implemented using the slf4j logging interface. As underlying logging framework, log4j is used. We also provide logback configuration files and pass them to the JVM's as properties. Users willing to use logback instead of log4j can just exclude log4j (or delete it from the lib/ folder).
+The logging in Flink is implemented using the slf4j logging interface. As underlying logging framework, log4j2 is used. We also provide logback configuration files and pass them to the JVM's as properties. Users willing to use logback instead of log4j2 can just exclude log4j2 (or delete it from the lib/ folder).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Configuring Log4j
+## Configuring Log4j2
 
-Log4j is controlled using property files. In Flink's case, the file is usually called `log4j.properties`. We pass the filename and location of this file using the `-Dlog4j.configuration=` parameter to the JVM.
+Log4j2 is controlled using property files. In Flink's case, the file is usually called `log4j.properties`. We pass the filename and location of this file using the `-Dlog4j.configurationFile=` parameter to the JVM.
 
 Flink ships with the following default properties files:
 
 - `log4j-cli.properties`: Used by the Flink command line client (e.g. `flink run`) (not code executed on the cluster)
 - `log4j-yarn-session.properties`: Used by the Flink command line client when starting a YARN session (`yarn-session.sh`)
 - `log4j.properties`: JobManager/Taskmanager logs (both standalone and YARN)
 
+### Compatibility with Log4j1
+
+Flink ships with the [Log4j API bridge](https://logging.apache.org/log4j/log4j-2.2/log4j-1.2-api/index.html), allowing existing applications that work against Log4j1 classes to continue working.
+
+If you have custom Log4j1 properties files or code that relies on Log4j1, please check out the official Log4j [compatibility](https://logging.apache.org/log4j/2.x/manual/compatibility.html) and [migration](https://logging.apache.org/log4j/2.x/manual/migration.html) guides.
+
+## Configuring Log4j1
+
+To use Flink with Log4j1 you must ensure that:
+- `org.apache.logging.log4j:log4j-core`, `org.apache.logging.log4j:log4j-slf4j-impl` and `org.apache.logging.log4j:log4j-1.2-api` are not on the classpath,
+- `log4j:log4j`, `org.slf4j:slf4j-log4j12`, `org.apache.logging.log4j:log4j-to-slf4j` and `org.apache.logging.log4j:log4j-api` are on the classpath.
+
+In the IDE this means you have to replace such dependencies defined in your pom, and possibly add exclusions on dependencies that transitively depend on them.
+
+For Flink distributions this means you have to
+- remove the `log4j-core`, `log4j-slf4j-impl` and `log4j-1.2-api` jars from the `lib` directory,
+- add the `log4j`, `slf4j-log4j12` and `log4j-to-slf4j` jars to the `lib` directory,
+- replace all log4j properties files in the `conf` directory with Log4j1-compliant versions.
+
 ## Configuring logback
 
 For users and developers alike it is important to control the logging framework.

diff --git a/docs/monitoring/logging.zh.md b/docs/monitoring/logging.zh.md
@@ -23,21 +23,40 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-The logging in Flink is implemented using the slf4j logging interface. As underlying logging framework, log4j is used. We also provide logback configuration files and pass them to the JVM's as properties. Users willing to use logback instead of log4j can just exclude log4j (or delete it from the lib/ folder).
+The logging in Flink is implemented using the slf4j logging interface. As underlying logging framework, log4j2 is used. We also provide logback configuration files and pass them to the JVM's as properties. Users willing to use logback instead of log4j2 can just exclude log4j2 (or delete it from the lib/ folder).
 
 * This will be replaced by the TOC
 {:toc}
 
-## Configuring Log4j
+## Configuring Log4j2
 
-Log4j is controlled using property files. In Flink's case, the file is usually called `log4j.properties`. We pass the filename and location of this file using the `-Dlog4j.configuration=` parameter to the JVM.
+Log4j2 is controlled using property files. In Flink's case, the file is usually called `log4j.properties`. We pass the filename and location of this file using the `-Dlog4j.configurationFile=` parameter to the JVM.
 
 Flink ships with the following default properties files:
 
 - `log4j-cli.properties`: Used by the Flink command line client (e.g. `flink run`) (not code executed on the cluster)
 - `log4j-yarn-session.properties`: Used by the Flink command line client when starting a YARN session (`yarn-session.sh`)
 - `log4j.properties`: JobManager/Taskmanager logs (both standalone and YARN)
 
+### Compatibility with Log4j1
+
+Flink ships with the [Log4j API bridge](https://logging.apache.org/log4j/log4j-2.2/log4j-1.2-api/index.html), allowing existing applications that work against Log4j1 classes to continue working.
+
+If you have custom Log4j1 properties files or code that relies on Log4j1, please check out the official Log4j [compatibility](https://logging.apache.org/log4j/2.x/manual/compatibility.html) and [migration](https://logging.apache.org/log4j/2.x/manual/migration.html) guides.
+
+## Configuring Log4j1
+
+To use Flink with Log4j1 you must ensure that:
+- `org.apache.logging.log4j:log4j-core`, `org.apache.logging.log4j:log4j-slf4j-impl` and `org.apache.logging.log4j:log4j-1.2-api` are not on the classpath,
+- `log4j:log4j`, `org.slf4j:slf4j-log4j12`, `org.apache.logging.log4j:log4j-to-slf4j` and `org.apache.logging.log4j:log4j-api` are on the classpath.
+
+In the IDE this means you have to replace such dependencies defined in your pom, and possibly add exclusions on dependencies that transitively depend on them.
+
+For Flink distributions this means you have to
+- remove the `log4j-core`, `log4j-slf4j-impl` and `log4j-1.2-api` jars from the `lib` directory,
+- add the `log4j`, `slf4j-log4j12` and `log4j-to-slf4j` jars to the `lib` directory,
+- replace all log4j properties files in the `conf` directory with Log4j1-compliant versions.
+
 ## Configuring logback
 
 For users and developers alike it is important to control the logging framework.

diff --git a/flink-connectors/flink-connector-elasticsearch-base/pom.xml b/flink-connectors/flink-connector-elasticsearch-base/pom.xml
@@ -147,39 +147,19 @@ under the License.
  <dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-api</artifactId>
- <version>2.7</version>
- <scope>test</scope>
+ <scope>provided</scope>
  </dependency>
 
  <dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-core</artifactId>
- <version>2.7</version>
  <scope>test</scope>
  </dependency>
 
  </dependencies>
 
  <build>
  <plugins>
- <!--
- For the tests, we need to exclude the Log4j2 to slf4j adapter dependency
- and let Elasticsearch directly use Log4j2, otherwise the embedded Elasticsearch node
- used in tests will fail to work.
-
- In other words, the connector jar is routing Elasticsearch 5.x's Log4j2 API's to SLF4J,
- but for the test builds, we still stick to directly using Log4j2.
- -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <classpathDependencyExcludes>
- <classpathDependencyExclude>org.apache.logging.log4j:log4j-to-slf4j</classpathDependencyExclude>
- </classpathDependencyExcludes>
- </configuration>
- </plugin>
-
  <plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-jar-plugin</artifactId>

diff --git a/flink-connectors/flink-connector-elasticsearch5/pom.xml b/flink-connectors/flink-connector-elasticsearch5/pom.xml
@@ -82,19 +82,6 @@ under the License.
  <version>${elasticsearch.version}</version>
  </dependency>
 
- <!--
- Elasticsearch 5.x uses Log4j2 and no longer detects logging implementations, making
- Log4j2 a strict dependency. The following is added so that the Log4j2 API in
- Elasticsearch 5.x is routed to SLF4J. This way, user projects can remain flexible
- in the logging implementation preferred.
- -->
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-to-slf4j</artifactId>
- <version>2.7</version>
- </dependency>
-
  <!-- test dependencies -->
 
  <dependency>
@@ -126,45 +113,10 @@ under the License.
  <scope>test</scope>
  </dependency>
 
- <!--
- Including Log4j2 dependencies for tests is required for the
- embedded Elasticsearch nodes used in tests to run correctly.
- -->
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-api</artifactId>
- <version>2.7</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-core</artifactId>
- <version>2.7</version>
- <scope>test</scope>
- </dependency>
-
  </dependencies>
 
  <build>
  <plugins>
- <!--
- For the tests, we need to exclude the Log4j2 to slf4j adapter dependency
- and let Elasticsearch directly use Log4j2, otherwise the embedded Elasticsearch node
- used in tests will fail to work.
-
- In other words, the connector jar is routing Elasticsearch 5.x's Log4j2 API's to SLF4J,
- but for the test builds, we still stick to directly using Log4j2.
- -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <classpathDependencyExcludes>
- <classpathDependencyExclude>org.apache.logging.log4j:log4j-to-slf4j</classpathDependencyExclude>
- </classpathDependencyExcludes>
- </configuration>
- </plugin>
  <plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
@@ -213,7 +165,7 @@ under the License.
  <excludes>
  <!-- keep flink classes as they are (exceptions as above) -->
  <exclude>org.apache.flink.**</exclude>
- <exclude>org.apache.log4j.**</exclude> <!-- provided -->
+ <exclude>org.apache.logging.log4j.**</exclude> <!-- provided -->
  </excludes>
  </relocation>
  <relocation>
@@ -270,17 +222,6 @@ under the License.
  <exclude>META-INF/maven/org.y*/**</exclude>
  </excludes>
  </filter>
- <!--
- Since we relocate the dependency, the paths specified in the providers file
- will be incorrect. We exclude it here, and re-package a new providers file
- with the correct re-located paths.
- -->
- <filter>
- <artifact>org.apache.logging.log4j:log4j-to-slf4j</artifact>
- <excludes>
- <exclude>META-INF/log4j-provider.properties</exclude>
- </excludes>
- </filter>
  <filter>
  <artifact>io.netty:netty</artifact>
  <excludes>

diff --git a/flink-connectors/flink-connector-elasticsearch5/src/main/resources/META-INF/NOTICE b/flink-connectors/flink-connector-elasticsearch5/src/main/resources/META-INF/NOTICE
@@ -29,8 +29,6 @@ This project bundles the following dependencies under the Apache Software Licens
 - org.apache.httpcomponents:httpclient:4.5.3
 - org.apache.httpcomponents:httpcore:4.4.6
 - org.apache.httpcomponents:httpcore-nio:4.4.5
-- org.apache.logging.log4j:log4j-api:2.7
-- org.apache.logging.log4j:log4j-to-slf4j:2.7
 - org.apache.lucene:lucene-analyzers-common:6.3.0
 - org.apache.lucene:lucene-backward-codecs:6.3.0
 - org.apache.lucene:lucene-core:6.3.0

diff --git a/...tors/flink-connector-elasticsearch5/src/main/resources/META-INF/log4j-provider.properties b/...tors/flink-connector-elasticsearch5/src/main/resources/META-INF/log4j-provider.properties
diff --git a/...st/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSinkITCase.java b/...st/java/org/apache/flink/streaming/connectors/elasticsearch5/ElasticsearchSinkITCase.java
@@ -34,10 +34,6 @@
 
 /**
  * IT cases for the {@link ElasticsearchSink}.
- *
- * <p>The Elasticsearch ITCases for 5.x CANNOT be executed in the IDE directly, since it is required that the
- * Log4J-to-SLF4J adapter dependency must be excluded from the test classpath for the Elasticsearch embedded
- * node used in the tests to work properly.
  */
 public class ElasticsearchSinkITCase extends ElasticsearchSinkTestBase<TransportClient, InetSocketAddress> {
 

diff --git a/flink-connectors/flink-connector-elasticsearch6/pom.xml b/flink-connectors/flink-connector-elasticsearch6/pom.xml
@@ -71,19 +71,6 @@ under the License.
  <version>${elasticsearch.version}</version>
  </dependency>
 
- <!--
- Elasticsearch 5.x uses Log4j2 and no longer detects logging implementations, making
- Log4j2 a strict dependency. The following is added so that the Log4j2 API in
- Elasticsearch 5.x is routed to SLF4J. This way, user projects can remain flexible
- in the logging implementation preferred.
- -->
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-to-slf4j</artifactId>
- <version>2.9.1</version>
- </dependency>
-
  <!-- Table ecosystem -->
  <!-- Projects depending on this project won't depend on flink-table-*. -->
  <dependency>
@@ -143,22 +130,10 @@ under the License.
  <scope>test</scope>
  </dependency>
 
- <!--
- Including Log4j2 dependencies for tests is required for the
- embedded Elasticsearch nodes used in tests to run correctly.
- -->
-
  <dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-api</artifactId>
- <version>2.9.1</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-core</artifactId>
- <version>2.9.1</version>
- <scope>test</scope>
+ <scope>provided</scope>
  </dependency>
 
  <!-- Elasticsearch table descriptor testing -->
@@ -179,30 +154,4 @@ under the License.
  </dependency>
 
  </dependencies>
-
- <build>
- <plugins>
- <!--
- For the tests, we need to exclude the Log4j2 to slf4j adapter dependency
- and let Elasticsearch directly use Log4j2, otherwise the embedded Elasticsearch node
- used in tests will fail to work.
-
- In other words, the connector jar is routing Elasticsearch 5.x's Log4j2 API's to SLF4J,
- but for the test builds, we still stick to directly using Log4j2.
- -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <systemPropertyVariables>
- <jna.nosys>true</jna.nosys>
- </systemPropertyVariables>
- <classpathDependencyExcludes>
- <classpathDependencyExclude>org.apache.logging.log4j:log4j-to-slf4j</classpathDependencyExclude>
- </classpathDependencyExcludes>
- </configuration>
- </plugin>
- </plugins>
- </build>
-
 </project>
diff --git a/...st/java/org/apache/flink/streaming/connectors/elasticsearch6/ElasticsearchSinkITCase.java b/...st/java/org/apache/flink/streaming/connectors/elasticsearch6/ElasticsearchSinkITCase.java
@@ -32,10 +32,6 @@
 
 /**
  * IT cases for the {@link ElasticsearchSink}.
- *
- * <p>The Elasticsearch ITCases for 6.x CANNOT be executed in the IDE directly, since it is required that the
- * Log4J-to-SLF4J adapter dependency must be excluded from the test classpath for the Elasticsearch embedded
- * node used in the tests to work properly.
  */
 public class ElasticsearchSinkITCase extends ElasticsearchSinkTestBase<RestHighLevelClient, HttpHost> {