[RLlib] Add on_checkpoint_loaded callback AND also store eval workers' policy_mapping_fn in algo state.
#40350
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: sven1977 <[email protected]>
sven1977
requested review from
gjoliver,
avnishn,
ArturNiederfahrenhorst,
smorad,
maxpumperla,
kouroshHakha and
krfricke
as code owners
Signed-off-by: sven1977 <[email protected]>
…e59_eval_policy_mapping_fn_not_checkpointed
Signed-off-by: sven1977 <[email protected]>
kouroshHakha
approved these changes
Oct 18, 2023
| @@ -74,6 +83,29 @@ def setUpClass(cls): | |||
| def tearDownClass(cls): | |||
| ray.shutdown() | |||
|
|
|||
| def test_on_init_and_checkpoint_loaded(self): | |||
| config = ( | |||
| PGConfig() | |||
There was a problem hiding this comment.
We are moving PG to rllib contrib. use PPO.
There was a problem hiding this comment.
Great catch! Will change.
|
some GPU restoration tests have not passed. |
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
…ot_checkpointed' into issue59_eval_policy_mapping_fn_not_checkpointed
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
jonathan-anyscale
pushed a commit
to jonathan-anyscale/ray
that referenced
this pull request
Oct 26, 2023
…rs' `policy_mapping_fn` in algo state. (ray-project#40350)
jonathan-anyscale
pushed a commit
to jonathan-anyscale/ray
that referenced
this pull request
Oct 26, 2023
…rs' `policy_mapping_fn` in algo state. (ray-project#40350)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes a problem with heavily customized eval WorkerSet setups, policy sets, and mapping functions.
Why are these changes needed?
When a user overrides the
on_algorithm_initcallback in order to setup special evaluation policies inside the evaluation worker set, including a new eval policy mapping function, then upon restoring this algorithm from a checkpoint, the evalpolicy_mapping_fninformation would be overridden by the mainpolicy_mapping_fn(b/c that one is the only one stored in the checkpoint).To solve this problem and to add additional handles for users with such complex customization needs, this PR:
policy_mapping_fnin the algorithm state, in case this mapping is different from the main mapping function.on_checkpoint_loaded()callback called after the Algorithm was restored from a checkpoint (Algorithm.load_checkpoint()has completed).Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.