Dynamic partition pruning #1102
Conversation
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. @@ Coverage Diff @@
## main #1102 +/- ##
==========================================
+ Coverage 81.48% 81.60% +0.11%
==========================================
Files 78 78
Lines 4509 4511 +2
Branches 828 828
==========================================
+ Hits 3674 3681 +7
+ Misses 653 644 -9
- Partials 182 186 +4
... and 1 file with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Waiting on https://github.com/rapidsai/ops/issues/2644 to resolve some datatype issues for the gpuCI tests. |
sarahyurick
requested review from
ayushdg,
charlesbluca,
galipremsagar and
jdye64
as code owners
dask_sql/context.py
Outdated
| if dask_config.get("sql.dynamic_partition_pruning"): | ||
| self.context.apply_dynamic_partition_pruning() | ||
|
|
There was a problem hiding this comment.
Sometime configs can be passed through the c.sql method as well https://github.com/dask-contrib/dask-sql/blob/main/dask_sql/context.py#L468.
Unless you need this during context initialization, it might make sense to set this variable somewhere in _get_ral before the query is parsed.
There was a problem hiding this comment.
Thanks! Done with 182fe8e - with this, DPP can be configured either globally (default True) or on a per-query basis with the config_options parameter.
WIP
Update: Should be ready to go, in combination with #1160 !