Katas - Convert task description from HTML to Markdown (apache#11736)

* Support ZetaSQL DATE type as a Beam LogicalType

* [BEAM-6733] Add pipeline option to flush bundle data before checkpointing

We had a couple of PRs in which we wanted to remove the buffering of bundle
output during checkpointing: apache#7940 apache#9652. Ultimately, we didn't merge any of
those because we weren't sure how the change would affect the checkpoint
performance.

As a better migration path, this introduces a pipeline option to change the
default, buffering bundle output during checkpointing, to finishing the bundle
and flushing all data before checkpointing.

* Remove all answer placeholder checks as they can be confusing at times for some learners

* Update course in Stepik

* [BEAM-10018] Fix timestamps in windowing kata

In this Kata, the timestamp was calculated from time objects, and converted to a
timestamp in the local timezone. Thus, the results of the test depended on the
configuration of the local timezone in the running system.

The tests were hardcoded with a timezone different to mine, and thus I always
failed to pass this Kata. The changes in this commit change the type in Event to
be a datetime, the timestamps are set in UTC, and the output in the tests is
hardcoded in UTC too. This should ensure that the kata works regardless the
timezone configured in the system running the kata.

* [BEAM-10018] Kata failing due to failed parsing

Parsing the timestamps as strings using fromisoformat was failing, and the Kata
failed silently regardless the code written in the boxes.

This change sets the same timestamps, with UTC timezone, without parsing
strings.

* Convert html task description to md for "Hello Beam" and "Core Transforms/Map"

* Remove unused import

* Add missing dependency

* Fix member variable name in Kata documentation

* Fix placeholder location

* Convert html task description to md for "Core Transforms" remaining lessons

* Convert html task description to md for "Common Transforms" lessons

* Convert html task description to md for remaining Python Katas lessons

* Convert html task description to md for most of Java Katas lessons

* Convert html task description to md for Java Katas "Common Transforms" lessons

* Convert html task description to md for Java Katas "Core Transforms" lessons

* [BEAM-2530] Implement Zeta SQL precommit compile tests and run on java 11 (apache#11692)

[BEAM-2530] Implement Zeta SQL precommit compile tests and run on java 11

* Python3 fix - convert dict.keys() to list before indexing (apache#11733)

* Updates google-apitools and httplib2 (apache#11726)

* [BEAM-9964] Update CHANGES.md  (apache#11743)

Co-authored-by: Omar Ismail <[email protected]>

* [BEAM-9577] Artifact v2 support for uber jars. (apache#11708)

* Adds a "filesystem" for artifacts placed on the classpath (e.g. within the uberjar).
* Updates the flink and spark uberjars to use artifact staging v2, leveraging the above filesystem.

* Populate all SpannerIO batching parameters in display data.

Add all the grouping/batching parameters in SpannerIO
populateDisplayData().

* Fix capitalization, clarify descriptions

* fix capitalization, clarify description Grouped

* Refactor to extract single method for popuplating displayData

* Convert html task description to md for "Hello Beam" and "Core Transforms/Map"

* Convert html task description to md for "Core Transforms" remaining lessons

* Convert html task description to md for "Common Transforms" lessons

* Convert html task description to md for remaining Python Katas lessons

* Convert html task description to md for most of Java Katas lessons

* Convert html task description to md for Java Katas "Common Transforms" lessons

* Convert html task description to md for Java Katas "Core Transforms" lessons

* Resolve merge conflict

* Update Python Katas on Stepik

* Update Beam Katas Java on Stepik

Co-authored-by: Yueyang Qiu <[email protected]>
Co-authored-by: Maximilian Michels <[email protected]>
Co-authored-by: Israel Herraiz <[email protected]>
Co-authored-by: pawelpasterz <[email protected]>
Co-authored-by: Chamikara Jayalath <[email protected]>
Co-authored-by: tvalentyn <[email protected]>
Co-authored-by: Pablo <[email protected]>
Co-authored-by: omarismail94 <[email protected]>
Co-authored-by: Omar Ismail <[email protected]>
Co-authored-by: Andrew Pilloud <[email protected]>
Co-authored-by: Robert Bradshaw <[email protected]>
Co-authored-by: nielm <[email protected]>
Co-authored-by: Brian Hulette <[email protected]>
Co-authored-by: Brian Hulette <[email protected]>

learning/katas/java/Common Transforms/Aggregation/Count/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076163
-update_date: Fri, 07 Feb 2020 14:07:39 UTC
+update_date: Tue, 19 May 2020 07:02:13 UTC

learning/katas/java/Common Transforms/Aggregation/Count/task.html → learning/katas/java/Common Transforms/Aggregation/Count/task.md

@@ -16,14 +16,12 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Aggregation - Count</h2>
-<p>
- <b>Kata:</b> Count the number of elements from an input.
-</p>
-<br>
+Aggregation - Count
+-------------------
+
+**Kata:** Count the number of elements from an input.
+
 <div class="hint">
  Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html">
  Count</a>.
 </div>
-</html>

learning/katas/java/Common Transforms/Aggregation/Max/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076167
-update_date: Fri, 07 Feb 2020 14:07:47 UTC
+update_date: Tue, 19 May 2020 07:02:25 UTC

learning/katas/java/Common Transforms/Aggregation/Max/task.html → learning/katas/java/Common Transforms/Aggregation/Max/task.md

@@ -16,14 +16,12 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Aggregation - Max</h2>
-<p>
- <b>Kata:</b> Compute the maximum of the elements from an input.
-</p>
-<br>
+Aggregation - Max
+-----------------
+
+**Kata:** Compute the maximum of the elements from an input.
+
 <div class="hint">
  Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Max.html">
  Max</a>.
 </div>
-</html>

learning/katas/java/Common Transforms/Aggregation/Mean/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076165
-update_date: Fri, 07 Feb 2020 14:07:43 UTC
+update_date: Tue, 19 May 2020 07:02:19 UTC

learning/katas/java/Common Transforms/Aggregation/Mean/task.html → learning/katas/java/Common Transforms/Aggregation/Mean/task.md

@@ -16,14 +16,12 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Aggregation - Mean</h2>
-<p>
- <b>Kata:</b> Compute the mean/average of all elements from an input.
-</p>
-<br>
+Aggregation - Mean
+------------------
+
+**Kata:** Compute the mean/average of all elements from an input.
+
 <div class="hint">
  Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Mean.html">
  Mean</a>.
 </div>
-</html>

learning/katas/java/Common Transforms/Aggregation/Min/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076166
-update_date: Fri, 07 Feb 2020 14:07:45 UTC
+update_date: Tue, 19 May 2020 07:02:22 UTC

learning/katas/java/Common Transforms/Aggregation/Min/task.html → learning/katas/java/Common Transforms/Aggregation/Min/task.md

@@ -16,14 +16,12 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Aggregation - Min</h2>
-<p>
- <b>Kata:</b> Compute the minimum of the elements from an input.
-</p>
-<br>
+Aggregation - Min
+-----------------
+
+**Kata:** Compute the minimum of the elements from an input.
+
 <div class="hint">
  Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Min.html">
  Min</a>.
 </div>
-</html>

learning/katas/java/Common Transforms/Aggregation/Sum/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076164
-update_date: Fri, 07 Feb 2020 14:07:41 UTC
+update_date: Tue, 19 May 2020 07:02:16 UTC

learning/katas/java/Common Transforms/Aggregation/Sum/task.html → learning/katas/java/Common Transforms/Aggregation/Sum/task.md

@@ -16,14 +16,12 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Aggregation - Sum</h2>
-<p>
- <b>Kata:</b> Compute the sum of all elements from an input.
-</p>
-<br>
+Aggregation - Sum
+-----------------
+
+**Kata:** Compute the sum of all elements from an input.
+
 <div class="hint">
  Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Sum.html">
  Sum</a>.
 </div>
-</html>

learning/katas/java/Common Transforms/Filter/Filter/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076162
-update_date: Fri, 07 Feb 2020 14:07:31 UTC
+update_date: Tue, 19 May 2020 07:02:10 UTC

learning/katas/java/Common Transforms/Filter/Filter/task.html → learning/katas/java/Common Transforms/Filter/Filter/task.md

@@ -16,19 +16,15 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Filter</h2>
-<p>
- The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.
-</p>
-<p>
- <b>Kata:</b> Implement a filter function that filters out the odd numbers by using
- <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html">
- Filter</a>.
-</p>
-<br>
+Filter
+------
+
+The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.
+
+**Kata:** Implement a filter function that filters out the odd numbers by using 
+[Filter](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html).
+
 <div class="hint">
  Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html">
  Filter.by(...)</a>.
 </div>
-</html>

learning/katas/java/Common Transforms/Filter/ParDo/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076161
-update_date: Fri, 07 Feb 2020 14:07:30 UTC
+update_date: Tue, 19 May 2020 07:02:07 UTC

learning/katas/java/Common Transforms/Filter/ParDo/task.html → learning/katas/java/Common Transforms/Filter/ParDo/task.md

@@ -16,18 +16,15 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Filter using ParDo</h2>
-<p>
- <b>Kata:</b> Implement a filter function that filters out the even numbers by using
- <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html">
- DoFn</a>.
-</p>
-<br>
+Filter using ParDo
+------------------
+
+**Kata:** Implement a filter function that filters out the even numbers by using 
+[DoFn](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html).
+
 <div class="hint">
  Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">
  ParDo</a> with
  <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html">
  DoFn</a> and only output the intended element.
 </div>
-</html>

learning/katas/java/Common Transforms/WithKeys/WithKeys/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076168
-update_date: Fri, 07 Feb 2020 14:07:53 UTC
+update_date: Tue, 19 May 2020 07:02:28 UTC

learning/katas/java/Common Transforms/WithKeys/WithKeys/task.html → learning/katas/java/Common Transforms/WithKeys/WithKeys/task.md

@@ -16,19 +16,18 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>WithKeys</h2>
-<p>
- <b>Kata:</b> Convert each fruit name into a KV of its first letter and itself, e.g.
- <code>apple => KV.of("a", "apple")</code>
-</p>
-<br>
+WithKeys
+--------
+
+**Kata:** Convert each fruit name into a KV of its first letter and itself, e.g. 
+`apple => KV.of("a", "apple")`
+
 <div class="hint">
  Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithKeys.html">
  WithKeys</a>.
 </div>
+
 <div class="hint">
  If using a lambda in Java 8, <code>withKeyType(TypeDescriptor)</code> must be called on the
  result PTransform.
 </div>
-</html>

learning/katas/java/Core Transforms/Branching/Branching/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076188
-update_date: Fri, 07 Feb 2020 14:28:13 UTC
+update_date: Tue, 19 May 2020 07:01:56 UTC

learning/katas/java/Core Transforms/Branching/Branching/task.html → learning/katas/java/Core Transforms/Branching/Branching/task.md

@@ -16,20 +16,17 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Branching</h2>
-<p>
- You can use the same PCollection as input for multiple transforms without consuming the input
- or altering it.
-</p>
-<p>
- <b>Kata:</b> Branch out the numbers to two different transforms: one transform is multiplying
- each number by 5 and the other transform is multiplying each number by 10.
-</p>
-<br>
+Branching
+---------
+
+You can use the same PCollection as input for multiple transforms without consuming the input or 
+altering it.
+
+**Kata:** Branch out the numbers to two different transforms: one transform is multiplying each 
+number by 5 and the other transform is multiplying each number by 10.
+
 <div class="hint">
  Refer to the Beam Design Your Pipeline Guide
  <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/#multiple-transforms-process-the-same-pcollection">
  "Multiple transforms process the same PCollection"</a> section for more information.
 </div>
-</html>

learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076151
-update_date: Fri, 07 Feb 2020 14:06:21 UTC
+update_date: Tue, 19 May 2020 07:08:21 UTC

learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task.html → learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task.md

@@ -16,19 +16,17 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>CoGroupByKey</h2>
-<p>
- CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
- key type.
-</p>
-<p>
- <b>Kata:</b> Implement a
- <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html">
- CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces
- the toString() representation of the WordsAlphabet model.
-</p>
-<br>
+CoGroupByKey
+------------
+
+CoGroupByKey performs a relational join of two or more key/value PCollections that have the same 
+key type.
+
+**Kata:** Implement a 
+[CoGroupByKey](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html) 
+transform that join words by its first alphabetical letter, and then produces the toString() 
+representation of the WordsAlphabet model.
+
 <div class="hint">
  Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html">
  CoGroupByKey</a>,
@@ -37,9 +35,9 @@ <h2>CoGroupByKey</h2>
  <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGbkResult.html">
  CoGbkResult</a>.
 </div>
+
 <div class="hint">
  Refer to the Beam Programming Guide
  <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
  "CoGroupByKey"</a> section for more information.
 </div>
-</html>

learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076154
-update_date: Fri, 07 Feb 2020 14:34:40 UTC
+update_date: Tue, 19 May 2020 07:01:38 UTC

learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task.html → learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task.md

@@ -16,28 +16,25 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Combine - BinaryCombineFn Lambda</h2>
-<p>
- BinaryCombineFn is used for implementing combiners that are more easily expressed as binary
- operations.
-</p>
-<p>
- Since Beam v2.13.0, you can also use lambda or method reference in order to create the
- BinaryCombineFn.
-</p>
-<p>
- <b>Kata:</b> Implement the summation of BigInteger using lambda or method reference.
-</p>
-<br>
+Combine - BinaryCombineFn Lambda
+--------------------------------
+
+BinaryCombineFn is used for implementing combiners that are more easily expressed as binary 
+operations.
+
+Since Beam v2.13.0, you can also use lambda or method reference in order to create the 
+BinaryCombineFn.
+
+**Kata:** Implement the summation of BigInteger using lambda or method reference.
+
 <div class="hint">
  Refer to
  <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableBiFunction.html">
  SerializableBiFunction</a>.
 </div>
+
 <div class="hint">
  Refer to the Beam Programming Guide
  <a href="https://beam.apache.org/documentation/programming-guide/#combine">
  "Combine"</a> section for more information.
 </div>
-</html>

learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076153
-update_date: Fri, 07 Feb 2020 14:34:37 UTC
+update_date: Tue, 19 May 2020 07:01:35 UTC

learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task.html → learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task.md

@@ -16,35 +16,30 @@
  ~ limitations under the License.
  -->
 
-<html>
-<h2>Combine - BinaryCombineFn</h2>
-<p>
- Combine is a Beam transform for combining collections of elements or values in your data.
- When you apply a Combine transform, you must provide the function that contains the logic for
- combining the elements or values. The combining function should be commutative and associative,
- as the function is not necessarily invoked exactly once on all values with a given key. Because
- the input data (including the value collection) may be distributed across multiple workers, the
- combining function might be called multiple times to perform partial combining on subsets of
- the value collection.
-</p>
-<p>
- BinaryCombineFn is used for implementing combiners that are more easily expressed as binary
- operations.
-</p>
-<p>
- <b>Kata:</b> Implement the summation of BigInteger using
- <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">
- Combine.BinaryCombineFn</a>.
-</p>
-<br>
+Combine - BinaryCombineFn
+-------------------------
+
+Combine is a Beam transform for combining collections of elements or values in your data. When you 
+apply a Combine transform, you must provide the function that contains the logic for combining the 
+elements or values. The combining function should be commutative and associative, as the function 
+is not necessarily invoked exactly once on all values with a given key. Because the input data 
+(including the value collection) may be distributed across multiple workers, the combining function 
+might be called multiple times to perform partial combining on subsets of the value collection.
+
+BinaryCombineFn is used for implementing combiners that are more easily expressed as binary 
+operations.
+
+**Kata:** Implement the summation of BigInteger using 
+[Combine.BinaryCombineFn](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html).
+
 <div class="hint">
  Extend the
  <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">
  Combine.BinaryCombineFn</a> class that counts the sum of the number.
 </div>
+
 <div class="hint">
  Refer to the Beam Programming Guide
  <a href="https://beam.apache.org/documentation/programming-guide/#combine">
  "Combine"</a> section for more information.
 </div>
-</html>

learning/katas/java/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml

@@ -1,2 +1,2 @@
 id: 1076155
-update_date: Fri, 07 Feb 2020 14:34:43 UTC
+update_date: Tue, 19 May 2020 07:01:41 UTC