Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log whenever we sleep #129197

Closed
wants to merge 2 commits into from
Closed

Log whenever we sleep #129197

wants to merge 2 commits into from

Conversation

c-p-i-o
Copy link
Contributor

@c-p-i-o c-p-i-o commented Jun 21, 2024

Stack from ghstack (oldest at bottom):

Summary:
Log whenever we sleep for heartbeatTimeout.
Useful for debugging stuck jobs.
This will eventually turn into a metric.

Test Plan:
none.

Reviewers:

Subscribers:

Tasks:

Tags:

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k

[ghstack-poisoned]
Copy link

pytorch-bot bot commented Jun 21, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129197

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 01e07ce with merge base d4022b4 (image):

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels Jun 21, 2024
@c-p-i-o c-p-i-o self-assigned this Jun 21, 2024
@c-p-i-o c-p-i-o requested review from d4l3k and wconstab June 21, 2024 01:33
torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp Outdated Show resolved Hide resolved
[ghstack-poisoned]
c-p-i-o added a commit that referenced this pull request Jun 21, 2024
Summary:
Log whenever we sleep for heartbeatTimeout.
Useful for debugging stuck jobs.
This will eventually turn into a metric.

Test Plan:
none.

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: 3d73602397b9c3a59b651a6511d7b614578bc3cc
Pull Request resolved: #129197
@c-p-i-o
Copy link
Contributor Author

c-p-i-o commented Jun 21, 2024

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 21, 2024
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

@c-p-i-o
Copy link
Contributor Author

c-p-i-o commented Jun 25, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the gh/c-p-i-o/20/head branch July 26, 2024 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants