Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Enable AOT inductor FP32 accuracy test for CPU #129040

Closed
wants to merge 1 commit into from
Closed

[CI] Enable AOT inductor FP32 accuracy test for CPU #129040

wants to merge 1 commit into from

Conversation

WeizhuoZhang-intel
Copy link
Contributor

@WeizhuoZhang-intel WeizhuoZhang-intel commented Jun 19, 2024 •
edited
Loading

This PR enabled AOT inductor backend FP32 accuracy check for CPU in CI workflow, which could catch AOT inductor issue at early stage.

Test Time cost:

Suite Precision Time cost
Huggingface FP32 1h12m
Timm models FP32 1h32m
Torchbench FP32 1h40m

cc @ezyang @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng

@WeizhuoZhang-intel WeizhuoZhang-intel requested a review from a team as a code owner June 19, 2024 05:10
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 19, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129040

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 147b695 with merge base b8398b7 (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@WeizhuoZhang-intel
Copy link
Contributor Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Tried to rebase and push PR #129040, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 19, 2024

Please seek CI approval before scheduling CIFlow labels

@WeizhuoZhang-intel
Copy link
Contributor Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased weizhuo/aot_inductor_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout weizhuo/aot_inductor_ci && git pull --rebase)

chuanqi129
chuanqi129 reviewed Jun 20, 2024
View reviewed changes
.github/workflows/inductor.yml Outdated
@@ -174,6 +174,11 @@ jobs:
{ config: "dynamic_cpu_inductor_timm", shard: 2, num_shards: 2, runner: "linux.12xlarge" },
{ config: "dynamic_cpu_inductor_torchbench", shard: 1, num_shards: 2, runner: "linux.12xlarge" },
{ config: "dynamic_cpu_inductor_torchbench", shard: 2, num_shards: 2, runner: "linux.12xlarge" },
{ config: "aot_cpu_inductor_huggingface_freezing", shard: 1, num_shards: 1, runner: "linux.12xlarge" },
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's change the name to cpu_aot_inductor_xxx

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@WeizhuoZhang-intel
Copy link
Contributor Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased weizhuo/aot_inductor_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout weizhuo/aot_inductor_ci && git pull --rebase)

@WeizhuoZhang-intel
Copy link
Contributor Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased weizhuo/aot_inductor_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout weizhuo/aot_inductor_ci && git pull --rebase)

@WeizhuoZhang-intel
Copy link
Contributor Author

@pytorchbot rebase

@chuanqi129
Copy link
Collaborator

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Tried to rebase and push PR #129040, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

@chuanqi129
Copy link
Collaborator

@pytorchbot rebase -b main

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased weizhuo/aot_inductor_ci onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout weizhuo/aot_inductor_ci && git pull --rebase)

@chuanqi129
Copy link
Collaborator

@pytorchbot rebase -b viable/strict

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased weizhuo/aot_inductor_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout weizhuo/aot_inductor_ci && git pull --rebase)

@chuanqi129
Copy link
Collaborator

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 27, 2024
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 22 jobs have failed, first few of them are: inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 1, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench_freezing, 1, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench_freezing, 2, 2, linux.12xlarge), inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench_amp_freezing, 1, 2, linux.16xlarge.spr)

Details for Dev Infra team Raised by workflow job

@WeizhuoZhang-intel
Copy link
Contributor Author

@pytorchbot rebase -b viable/strict

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased weizhuo/aot_inductor_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout weizhuo/aot_inductor_ci && git pull --rebase)

@chuanqi129
Copy link
Collaborator

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

pytorchmergebot pushed a commit to khushi-411/pytorch that referenced this pull request Jul 2, 2024
@WeizhuoZhang-intel @pytorchmergebot
[CI] Enable AOT inductor FP32 accuracy test for CPU (pytorch#129040)
d731fe7 
This PR enabled AOT inductor backend FP32 accuracy check for CPU in CI workflow, which could catch AOT inductor issue at early stage.

**Test Time cost:**
| Suite       	| Precision 	| Time cost 	|
|-------------	|-----------	|-----------	|
| Huggingface 	| FP32      	|   1h12m   	|
| Timm models 	| FP32      	|   1h32m   	|
|  Torchbench 	| FP32      	|   1h40m   	|

Pull Request resolved: pytorch#129040
Approved by: https://github.com/chuanqi129, https://github.com/desertfire, https://github.com/malfet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@malfet malfet malfet approved these changes

@desertfire desertfire desertfire approved these changes

@chuanqi129 chuanqi129 chuanqi129 approved these changes

@chunyuan-w chunyuan-w Awaiting requested review from chunyuan-w

@jgong5 jgong5 Awaiting requested review from jgong5

Assignees
No one assigned
Labels
ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: dynamo oncall: pt2 open source release notes: releng release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@WeizhuoZhang-intel @pytorchmergebot @chuanqi129 @malfet @desertfire @pytorchbot @drisspg