Add parser option for parsing SQL numeric literals as decimal #4102
Conversation
andygrove
force-pushed
the
sql-decimals
branch
from
435e059
to
046da82
Compare
andygrove
changed the title
|
@kmitchener @viirya @Dandandan PTAL |
datafusion/sql/src/planner.rs
Outdated
| "SELECT 1, 1.0, 0.1, .1, 12.34", | ||
| "Projection: Int64(1), Decimal128(Some(10),2,1), Decimal128(Some(1),2,1), Decimal128(Some(1),1,1), Decimal128(Some(1234),4,2)\ |
There was a problem hiding this comment.
0.1 should be Decimal(1, 1), I think.
scala> sql("select 0.1")
res0: org.apache.spark.sql.DataFrame = [0.1: decimal(1,1)]
There was a problem hiding this comment.
we should try to use the min precision to represent the value in the decimal literal
| let number = n.parse::<i128>().map_err(|_| { | ||
| DataFusionError::from(ParserError(format!( | ||
| "Cannot parse {} as i128 when building decimal", | ||
| n | ||
| ))) | ||
| })?; | ||
| Ok(Expr::Literal(ScalarValue::Decimal128(Some(number), 38, 0))) |
There was a problem hiding this comment.
Can we get the minimum precision to handle this number?
scala> sql("select 10000000000000000000")
res10: org.apache.spark.sql.DataFrame = [10000000000000000000: decimal(20,0)]
There was a problem hiding this comment.
@viirya Could you take another look when you have time?
There was a problem hiding this comment.
I'd love to do it. Let me take another look.
andygrove
force-pushed
the
sql-decimals
branch
from
8f2e730
to
cd6c86b
Compare
datafusion/sql/src/planner.rs
Outdated
| } | ||
|
|
||
| /// Convert SQL data type to relational representation of data type | ||
| pub fn convert_data_type(sql_type: &SQLDataType) -> Result<DataType> { |
There was a problem hiding this comment.
Is this committed in accidentally?
There was a problem hiding this comment.
Not sure why this change adds this and convert_simple_data_type. Otherwise other change looks good to me.
There was a problem hiding this comment.
Yeah, this is from a merge conflict. I will fix.
|
Benchmark runs are scheduled for baseline = ebb24c5 and contender = 406c108. 406c108 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
|
🎉 |
Which issue does this PR close?
Part of #4072
Rationale for this change
Main motivation is to get benchmark query q6 returning the correct results but this is also the correct behavior for SQL
What changes are included in this PR?
Are there any user-facing changes?