Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

Multi-node checkpoint recovery fix #478

Merged
merged 23 commits into from
Jun 10, 2021
Merged

Multi-node checkpoint recovery fix #478

merged 23 commits into from
Jun 10, 2021

Conversation

melanibe
Copy link
Contributor

@melanibe melanibe commented Jun 7, 2021 •
edited
Loading

Fixes #473

@melanibe melanibe added the no changelog needed CHANGELOG.md does not need to be updated in this PR label Jun 10, 2021
@melanibe melanibe marked this pull request as ready for review June 10, 2021 13:08
@melanibe melanibe merged commit 267a197 into main Jun 10, 2021
@melanibe melanibe deleted the melanibe/fix-mulit-node-recovery branch June 10, 2021 13:44
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@ant0nsc ant0nsc ant0nsc approved these changes

@Shruthi42 Shruthi42 Shruthi42 approved these changes

Assignees
No one assigned
Labels
no changelog needed CHANGELOG.md does not need to be updated in this PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Download of recovery checkpoints should only happen on rank 0 in distributed training
3 participants
@melanibe @ant0nsc @Shruthi42