Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logger state dumping and restoring #13069

Open
wants to merge 25 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
65d8b6f
Add logger state dumping
ashleve May 18, 2022
71bee7b
Add logger state restoring
ashleve May 19, 2022
5ffa446
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 19, 2022
b2f1c16
Update log messages
ashleve May 19, 2022
1bfaa03
Comment out logger restoration
ashleve May 19, 2022
e14c8b3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 19, 2022
13258e4
Fix _call_loggers_load_state_dict
ashleve May 19, 2022
aae50e7
Merge branch 'master' into feature/logger-state-dumping
Borda Jun 21, 2022
cce4fbb
Merge branch 'master' into feature/logger-state-dumping
ashleve Jul 14, 2022
706bb7f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 14, 2022
0a63105
Add logger state_dict mock
ashleve Jul 14, 2022
e2696d3
Fix hooks tests
ashleve Jul 14, 2022
db674fb
fix hook test
ashleve Jul 14, 2022
d52bc69
Merge branch 'Lightning-AI:master' into feature/logger-state-dumping
ashleve Jul 15, 2022
278e26a
Fix test_hooks.py
ashleve Jul 15, 2022
c45a1a9
Merge branch 'feature/logger-state-dumping' of https://github.com/ash…
ashleve Jul 15, 2022
7f95a9c
update test_hooks.py
ashleve Jul 15, 2022
4caad5c
Uncomment restoring loggers
Jul 15, 2022
fb4d3dd
Fix test_model_checkpoint.py
ashleve Jul 15, 2022
6b2b7a7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 15, 2022
b94f459
Fix test_model_checkpoints.py
ashleve Jul 15, 2022
17ff116
add state key
awaelchli Nov 13, 2022
4ddcb97
Merge branch 'master' into feature/logger-state-dumping
awaelchli Nov 14, 2022
1c0e936
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 14, 2022
c5941df
Merge branch 'master' into feature/logger-state-dumping
Borda Dec 7, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update log messages
  • Loading branch information
ashleve committed May 19, 2022
commit b2f1c166c3052d160b15f4fbb1a1e008784849a0
4 changes: 2 additions & 2 deletions pytorch_lightning/trainer/trainer.py
Original file line number Diff line number Diff line change
Expand Up @@ -1116,7 +1116,7 @@ def _run(

# check if we should delay restoring checkpoint till later
if not self.strategy.restore_checkpoint_after_setup:
log.detail(f"{self.__class__.__name__}: restoring module and callbacks from checkpoint path: {ckpt_path}")
log.detail(f"{self.__class__.__name__}: restoring module, loggers and callbacks from checkpoint path: {ckpt_path}")
self._restore_modules_and_loggers_and_callbacks(ckpt_path)

log.detail(f"{self.__class__.__name__}: configuring sharded model")
Expand Down Expand Up @@ -1165,7 +1165,7 @@ def _run(
self._log_hyperparams()

if self.strategy.restore_checkpoint_after_setup:
log.detail(f"{self.__class__.__name__}: restoring module and callbacks from checkpoint path: {ckpt_path}")
log.detail(f"{self.__class__.__name__}: restoring module, loggers and callbacks from checkpoint path: {ckpt_path}")
self._restore_modules_and_loggers_and_callbacks(ckpt_path)

# restore optimizers, etc.
Expand Down