Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Enable amp accuracy check for inductor cpu #127758

Closed
wants to merge 5 commits into from
Closed

[CI] Enable amp accuracy check for inductor cpu #127758

wants to merge 5 commits into from

Conversation

DiweiSun
Copy link
Contributor

@DiweiSun DiweiSun commented Jun 3, 2024 •
edited by pytorch-bot bot
Loading

This is to enable inductor AMP accuracy check for on CPU in CI workflow to capture issue early. Three suites are included: timms, huggingface as well as torchbench.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 3, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127758

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (4 Unrelated Failures)

As of commit 76b2387 with merge base 7b7f357 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added module: dynamo release notes: releng release notes category labels Jun 3, 2024
@chuanqi129
Copy link
Collaborator

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Tried to rebase and push PR #127758, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

@chuanqi129
Copy link
Collaborator

@pytorchbot rebase -b main

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased molly/inducto_amp_accuracy onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout molly/inducto_amp_accuracy && git pull --rebase)

@DiweiSun
Copy link
Contributor Author

DiweiSun commented Jun 5, 2024

@pytorchbot rebase -b main

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased molly/inducto_amp_accuracy onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout molly/inducto_amp_accuracy && git pull --rebase)

@DiweiSun DiweiSun marked this pull request as ready for review June 6, 2024 05:11
@DiweiSun DiweiSun requested a review from a team as a code owner June 6, 2024 05:11
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 6, 2024

Please seek CI approval before scheduling CIFlow labels

@DiweiSun
Copy link
Contributor Author

DiweiSun commented Jun 6, 2024

This PR is from #125748 , which is closed by @lezcano due to too many people tagged.

@chuanqi129 chuanqi129 requested a review from lezcano June 6, 2024 14:10
@lezcano lezcano removed their request for review June 6, 2024 14:11
@janeyx99 janeyx99 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 6, 2024
@sanchitintel sanchitintel mentioned this pull request Jun 7, 2024
Closed
@chuanqi129 chuanqi129 requested a review from atalman June 11, 2024 09:17
@chuanqi129
Copy link
Collaborator

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@DiweiSun
Copy link
Contributor Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased molly/inducto_amp_accuracy onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout molly/inducto_amp_accuracy && git pull --rebase)

@DiweiSun
Copy link
Contributor Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased molly/inducto_amp_accuracy onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout molly/inducto_amp_accuracy && git pull --rebase)

@DiweiSun
Copy link
Contributor Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Tried to rebase and push PR #127758, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

desertfire
desertfire reviewed Jun 24, 2024
View reviewed changes
.github/workflows/inductor.yml
{ config: "cpu_inductor_timm_amp_freezing", shard: 1, num_shards: 2, runner: "linux.16xlarge.spr" },
{ config: "cpu_inductor_timm_amp_freezing", shard: 2, num_shards: 2, runner: "linux.16xlarge.spr" },
{ config: "cpu_inductor_torchbench_amp_freezing", shard: 1, num_shards: 2, runner: "linux.16xlarge.spr" },
{ config: "cpu_inductor_torchbench_amp_freezing", shard: 2, num_shards: 2, runner: "linux.16xlarge.spr" },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @huydhn , we have been growing this CPU test list quite a lot lately. I know CPU instances are usually cheaper, but still want to check with you if budget will be a concern.

@DiweiSun
Copy link
Contributor Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased molly/inducto_amp_accuracy onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout molly/inducto_amp_accuracy && git pull --rebase)

@chuanqi129
Copy link
Collaborator

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 25, 2024
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@desertfire desertfire desertfire approved these changes

@jgong5 jgong5 jgong5 approved these changes

@atalman atalman Awaiting requested review from atalman

Assignees
No one assigned
Labels
ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: dynamo open source release notes: releng release notes category topic: not user facing topic category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@DiweiSun @chuanqi129 @pytorchmergebot @desertfire @jgong5 @pytorchbot @janeyx99