Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-35479][SQL] Format PartitionFilters IN strings in scan nodes
### What changes were proposed in this pull request? This PR proposes to format strings correctly for `PushedFilters`. For example, `explain()` for a query below prints `v in (array('a'))` as `PushedFilters: [In(v, [WrappedArray(a)])]`; ``` scala> sql("create table t (v array<string>) using parquet") scala> sql("select * from t where v in (array('a'), null)").explain() == Physical Plan == *(1) Filter v#4 IN ([a],null) +- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN ([a],null)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t], PartitionFilters: [], PushedFilters: [In(v, [WrappedArray(a),null])], ReadSchema: struct<v:array<string>> ``` This PR makes `explain()` print it as `PushedFilters: [In(v, [[a]])]`; ``` scala> sql("select * from t where v in (array('a'), null)").explain() == Physical Plan == *(1) Filter v#4 IN ([a],null) +- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN ([a],null)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t], PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema: struct<v:array<string>> ``` NOTE: This PR includes a bugfix caused by apache#32577 (See the cloud-fan comment: https://github.com/apache/spark/pull/32577/files#r636108150). ### Why are the changes needed? To improve explain strings. ### Does this PR introduce _any_ user-facing change? Yes, this PR improves the explain strings for pushed-down filters. ### How was this patch tested? Added tests in `SQLQueryTestSuite`. Closes apache#32615 from maropu/ExplainPartitionFilters. Authored-by: Takeshi Yamamuro <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
- Loading branch information