Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(query): refactor hash join spill #15746

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

Dousir9
Copy link
Member

@Dousir9 Dousir9 commented Jun 6, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Code readability and maintainability

  1. New step transition mechanism: this PR divides hash join probe/build step into three types: SyncStep, AsyncStep, and Finish, ensuring that step transition occurs in the event and this PR redesigns step transition to make the step transition logic clearer and improve code readability and maintainability.
  2. Unify ProbeSpillHandler and BuildSpillHandler to HashJoinSpiller to improve code maintainability.
  3. Unify the spill logic of cross join and other types.
  4. Introduce HashTableType to reduce code.

Bug Fixed

  1. Fixed a corner case of left-related join when spill happens.
  2. Left Anti join can't probe first round when spill happens.

Performance

Improved the performance of partition.

Tests

  • TPC-H spill test
  • TPC-DS spill test
  • Wizard join spill test

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe): code maintainability

This change is Reviewable

@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Jun 6, 2024
@Dousir9 Dousir9 added the ci-cloud Build docker image for cloud test label Jun 7, 2024
Copy link
Contributor

github-actions bot commented Jun 7, 2024

Docker Image for PR

  • tag: pr-15746-9107967-1717765135

note: this image tag is only available for internal use,
please check the internal doc for more details.

@Dousir9 Dousir9 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jun 11, 2024
Copy link
Contributor

Docker Image for PR

  • tag: pr-15746-61aa91b-1718103773

note: this image tag is only available for internal use,
please check the internal doc for more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-cloud Build docker image for cloud test pr-refactor this PR changes the code base without new features or bugfix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant