Skip to content

Commit

Permalink
[SPARK-35479][SQL] Format PartitionFilters IN strings in scan nodes
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR proposes to format strings correctly for `PushedFilters`. For example, `explain()` for a query below prints `v in (array('a'))` as `PushedFilters: [In(v, [WrappedArray(a)])]`;

```
scala> sql("create table t (v array<string>) using parquet")
scala> sql("select * from t where v in (array('a'), null)").explain()
== Physical Plan ==
*(1) Filter v#4 IN ([a],null)
+- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN ([a],null)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t], PartitionFilters: [], PushedFilters: [In(v, [WrappedArray(a),null])], ReadSchema: struct<v:array<string>>
```
This PR makes `explain()` print it as `PushedFilters: [In(v, [[a]])]`;
```
scala> sql("select * from t where v in (array('a'), null)").explain()
== Physical Plan ==
*(1) Filter v#4 IN ([a],null)
+- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN ([a],null)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>
```
NOTE: This PR includes a bugfix caused by apache#32577 (See the cloud-fan comment: https://github.com/apache/spark/pull/32577/files#r636108150).

### Why are the changes needed?

To improve explain strings.

### Does this PR introduce _any_ user-facing change?

Yes, this PR improves the explain strings for pushed-down filters.

### How was this patch tested?

Added tests in `SQLQueryTestSuite`.

Closes apache#32615 from maropu/ExplainPartitionFilters.

Authored-by: Takeshi Yamamuro <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
  • Loading branch information
maropu authored and cloud-fan committed May 21, 2021
1 parent e1296ea commit 1a923f5
Show file tree
Hide file tree
Showing 4 changed files with 65 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -173,9 +173,14 @@ case class In(attribute: String, values: Array[Any]) extends Filter {
a == attribute && vs.length == values.length && vs.zip(values).forall(x => x._1 == x._2)
case _ => false
}
private def formatValue(v: Any): String = v match {
case null => "null"
case ar: Seq[Any] => ar.map(formatValue).mkString("[", ", ", "]")
case _ => v.toString
}
override def toString: String = {
// Sort elements for deterministic behaviours
s"In($attribute, [${values.map(_.toString).sorted.mkString(",")}])"
s"In($attribute, [${values.map(formatValue).sorted.mkString(",")}])"
}

override def references: Array[String] = Array(attribute) ++ values.flatMap(findReferences)
Expand Down
5 changes: 5 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/explain.sql
Original file line number Diff line number Diff line change
Expand Up @@ -124,3 +124,8 @@ DROP TABLE explain_temp1;
DROP TABLE explain_temp2;
DROP TABLE explain_temp3;
DROP TABLE explain_temp4;

-- SPARK-35479: Format PartitionFilters IN strings in scan nodes
CREATE table t(v array<string>) USING PARQUET;
EXPLAIN SELECT * FROM t WHERE v IN (array('a'), null);
DROP TABLE t;
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 24
-- Number of queries: 27


-- !query
Expand Down Expand Up @@ -1113,3 +1113,29 @@ DROP TABLE explain_temp4
struct<>
-- !query output



-- !query
CREATE table t(v array<string>) USING PARQUET
-- !query schema
struct<>
-- !query output



-- !query
EXPLAIN SELECT * FROM t WHERE v IN (array('a'), null)
-- !query schema
struct<plan:string>
-- !query output
== Physical Plan ==
*Filter v#x IN ([a],null)
+- FileScan parquet default.t[v#x] Batched: false, DataFilters: [v#x IN ([a],null)], Format: Parquet, Location [not included in comparison]/{warehouse_dir}/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>


-- !query
DROP TABLE t
-- !query schema
struct<>
-- !query output

28 changes: 27 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/explain.sql.out
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 24
-- Number of queries: 27


-- !query
Expand Down Expand Up @@ -1055,3 +1055,29 @@ DROP TABLE explain_temp4
struct<>
-- !query output



-- !query
CREATE table t(v array<string>) USING PARQUET
-- !query schema
struct<>
-- !query output



-- !query
EXPLAIN SELECT * FROM t WHERE v IN (array('a'), null)
-- !query schema
struct<plan:string>
-- !query output
== Physical Plan ==
*Filter v#x IN ([a],null)
+- FileScan parquet default.t[v#x] Batched: false, DataFilters: [v#x IN ([a],null)], Format: Parquet, Location [not included in comparison]/{warehouse_dir}/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>>


-- !query
DROP TABLE t
-- !query schema
struct<>
-- !query output

0 comments on commit 1a923f5

Please sign in to comment.