Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix most benchmark query regressions #4037

Closed
wants to merge 5 commits into from

Conversation

andygrove
Copy link
Member

@andygrove andygrove commented Oct 30, 2022

Which issue does this PR close?

Closes #4026
Closes #4025
Closes #4023

Rationale for this change

We cannot use decimal types in the benchmark until they are fully implemented. Overflows are causing queries to return very incorrect results and take longer to run because they return too many rows in some cases.

What changes are included in this PR?

  • Revert to Float64 in benchmark schema
  • Update tests to tolerate small rounding differences as a temporary measure until we move to decimals

Are there any user-facing changes?

No

Comment on lines +43 to +47
fn decimal_type(_p: u8, _s: u8) -> DataType {
// TODO use decimal_type(p, s) once Decimal is fully supported
// https://github.com/apache/arrow-datafusion/issues/3523
DataType::Float64
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we're ready to try decimal again, we just need to update this one function

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also add a CLI option to switch between float and decimal

@github-actions github-actions bot added the core Core DataFusion crate label Oct 30, 2022
@andygrove andygrove marked this pull request as ready for review October 30, 2022 19:39
@github-actions github-actions bot removed the core Core DataFusion crate label Oct 30, 2022
Copy link
Contributor

@tustvold tustvold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me, the decimal support is still work in progress.

FWIW these is ongoing work to properly support decimal upstream, if anyone is willing to help out

Copy link
Contributor

@Dandandan Dandandan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me 👍

@andygrove andygrove marked this pull request as draft October 31, 2022 14:25
@andygrove
Copy link
Member Author

Closing this in favor of #4044

@andygrove andygrove closed this Oct 31, 2022
@andygrove andygrove deleted the benchmark-use-float64 branch January 27, 2023 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants