Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-35935][table] Fix RTAS not supporting LIMIT #25185

Merged
merged 2 commits into from
Aug 15, 2024

Conversation

xuyangzhong
Copy link
Contributor

What is the purpose of the change

When using SqlReplaceTableAsConverter to convert SqlNode, SqlReplaceTableAsConverter wrongly used the original query asQuerySqlNode. Actually it should use the validated SqlNode.

Brief change log

  • Use validated SqlNode when converting RTAS
  • Add full tests

Verifying this change

Many tests (including UT and IT) have been added to verify this pr.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented?

@flinkbot
Copy link
Collaborator

flinkbot commented Aug 9, 2024

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Copy link
Contributor

@lincoln-lil lincoln-lil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xuyangzhong Thanks for fixing this! The only question from my side is the reason why single parallelism must set for these tests?

QueryOperation query =
new PlannerQueryOperation(
context.toRelRoot(asQuerySqlNode).project(),
context.toRelRoot(validated).project(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

@@ -78,6 +78,29 @@ void testReplaceTableAS() throws Exception {
verifyCatalogTable(expectCatalogTable, getCatalogTable("target"));
}

@Test
void testReplaceTableASWithLimit() throws Exception {
env().setParallelism(1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why must be single parallelism?

@@ -108,6 +108,27 @@ void testCreateOrReplaceTableAS() throws Exception {
verifyCatalogTable(expectCatalogTable, getCatalogTable("target"));
}

@Test
void testCreateOrReplaceTableASWithLimit() throws Exception {
env().setParallelism(1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@xuyangzhong
Copy link
Contributor Author

Hi, @lincoln-lil . The reason is: The plan will be optimized to source -> exchange (single) -> limit. Theoretically, if the source outputs data with multi parallelism, it could result in variations in the query results, leading to unstable tests. To mitigate this risk, I am enforcing single parallelism in this case.

@lincoln-lil
Copy link
Contributor

@xuyangzhong So at least we can use a sort clause for batch tests to get the stable result?

@xuyangzhong
Copy link
Contributor Author

@lincoln-lil You're right. I have replaced all limit with sort limit in IT cases.

Copy link
Contributor

@lincoln-lil lincoln-lil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xuyangzhong LGTM!

@lincoln-lil lincoln-lil merged commit 729b8b8 into apache:master Aug 15, 2024
xuyangzhong added a commit to xuyangzhong/flink that referenced this pull request Aug 15, 2024
xuyangzhong added a commit to xuyangzhong/flink that referenced this pull request Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants