-
Notifications
You must be signed in to change notification settings - Fork 5.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[train] Storage: Call callbacks with new checkpoint #38456
[train] Storage: Call callbacks with new checkpoint #38456
Conversation
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! One suggestion to fix the test.
@@ -1074,7 +1074,7 @@ def clear_checkpoint(self): | |||
self.temporary_state.restoring_from = None | |||
self.run_metadata.invalidate_cache() | |||
|
|||
def on_checkpoint(self, checkpoint: _TrackedCheckpoint): | |||
def on_checkpoint(self, checkpoint: Union[_TrackedCheckpoint, _TrainingResult]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh I haven't been so on top of updating all of these 😄
Signed-off-by: Kai Fricke <[email protected]>
…checkpoint # Conflicts: # python/ray/tune/execution/tune_controller.py
saved_iter = "?" | ||
if trial.last_result and TRAINING_ITERATION in trial.last_result: | ||
saved_iter = trial.last_result[TRAINING_ITERATION] | ||
|
||
self._start_block(f"trial_{trial}_result_{saved_iter}") | ||
|
||
if isinstance(checkpoint, Checkpoint): | ||
loc = f"({checkpoint.filesystem.type_name}){checkpoint.path}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We currently have no way to restore the actual location URI, so this is the next best thing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add an example of what this looks like in the pr description?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, lgtm. Only left some nits.
saved_iter = "?" | ||
if trial.last_result and TRAINING_ITERATION in trial.last_result: | ||
saved_iter = trial.last_result[TRAINING_ITERATION] | ||
|
||
self._start_block(f"trial_{trial}_result_{saved_iter}") | ||
|
||
if isinstance(checkpoint, Checkpoint): | ||
loc = f"({checkpoint.filesystem.type_name}){checkpoint.path}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add an example of what this looks like in the pr description?
Good to merge, but just in case you want to address the nits. |
Signed-off-by: Kai Fricke <[email protected]>
Adjust checkpoint interface to accept new-style checkpoint objects instead. In progress reporter output: ``` Training saved a checkpoint for iteration 4 at: (s3)bucket/foo/bar ``` Signed-off-by: Kai Fricke <[email protected]> Signed-off-by: e428265 <[email protected]>
Adjust checkpoint interface to accept new-style checkpoint objects instead. In progress reporter output: ``` Training saved a checkpoint for iteration 4 at: (s3)bucket/foo/bar ``` Signed-off-by: Kai Fricke <[email protected]> Signed-off-by: Victor <[email protected]>
Why are these changes needed?
Adjust checkpoint interface to accept new-style checkpoint objects instead.
In progress reporter output:
Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.