Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[air] Serialize additional files in dict checkpoints turned dir checkpoints #26351

Merged
merged 7 commits into from
Jul 8, 2022
Merged

[air] Serialize additional files in dict checkpoints turned dir checkpoints #26351

merged 7 commits into from
Jul 8, 2022

Conversation

krfricke
Copy link
Contributor

@krfricke krfricke commented Jul 7, 2022 •
edited
Loading

Why are these changes needed?

With this PR, files put into directory checkpoints that were dict checkpoints will be serialized and retained when a subsequent to_dict() is called. This is to enable storing additional files, as e.g. needed by Ray Tune.

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@krfricke krfricke marked this pull request as ready for review July 7, 2022 15:04
@krfricke krfricke mentioned this pull request Jul 7, 2022
Merged
6 tasks
Yard1
Yard1 reviewed Jul 7, 2022
View reviewed changes
python/ray/air/checkpoint.py Show resolved Hide resolved
python/ray/air/checkpoint.py Show resolved Hide resolved
python/ray/air/tests/test_checkpoints.py Show resolved Hide resolved
Kai Fricke added 3 commits July 7, 2022 23:18
Updated tests, docstring
c3d4df3 
Signed-off-by: Kai Fricke <[email protected]>
Test with nested additional file
61b29dd 
Signed-off-by: Kai Fricke <[email protected]>
nit
d76a502 
Signed-off-by: Kai Fricke <[email protected]>
@krfricke
Copy link
Contributor Author

krfricke commented Jul 7, 2022

@xwjiang2010 @Yard1 updated, ptal

xwjiang2010
xwjiang2010 approved these changes Jul 7, 2022
View reviewed changes
Copy link
Contributor

@xwjiang2010 xwjiang2010 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kai!

@krfricke krfricke merged commit 86b9b4b into ray-project:master Jul 8, 2022
@krfricke krfricke deleted the air/checkpoint-separate branch July 8, 2022 09:03
krfricke added a commit that referenced this pull request Jul 8, 2022
@krfricke
[tune] Use Checkpoint.to_bytes() for store_to_object (#25805)
e1a7efe 
We currently use our own serialization to ship checkpoints as objects. Instead we should use the Checkpoint class. This PR also adds support to create results from checkpoints pointing to object references.

Depends on #26351

Signed-off-by: Kai Fricke <[email protected]>
truelegion47 pushed a commit to truelegion47/ray that referenced this pull request Jul 9, 2022
Merge branch 'master' into ISSUE-24869
7744427 
* master: (42 commits)
  [dashboard][2/2] Add endpoints to dashboard and dashboard_agent for liveness check of raylet and gcs (ray-project#26408)
  [Doc] Fix docs feedback button (ray-project#26402)
  [core][1/2] Improve liveness check in GCS  (ray-project#26405)
  [RLlib] Checkpoint and restore connectors. (ray-project#26253)
  [Workflow] Minor refactoring of workflow exceptions (ray-project#26398)
  [workflow] Workflow queue (ray-project#24697)
  [RLlib] Minor simplification of code. (ray-project#26312)
  [AIR] Update TensorflowPredictor to new API (ray-project#26215)
  [RLlib] Make Dataset reader default reader and enable CRR to use dataset (ray-project#26304)
  [runtime_env] [doc] Remove outdated info about "isolated" environment (ray-project#26314)
  [Doc] Fix rate-the-docs plugin (ray-project#26384)
  [Docs] [Serve] Has a consistent landing page style (ray-project#26029)
  [dashboard] Add `RAY_CLUSTER_ACTIVITY_HOOK` to `/api/component_activities` (ray-project#26297)
  [tune] Use `Checkpoint.to_bytes()` for store_to_object (ray-project#25805)
  [tune] Fix `SyncerCallback` having a size limit (ray-project#26371)
  [air] Serialize additional files in dict checkpoints turned dir checkpoints (ray-project#26351)
  [Docs] Add "rate the docs" plugin for feedback on docs (ray-project#26330)
  [Doc] Fix actor example (ray-project#26381)
  Set RAY_USAGE_STATS_EXTRA_TAGS for release tests (ray-project#26366)
  [Datasets] Update docs for drop_columns and fix typos (ray-project#26317)
  ...
Stefan-1313 pushed a commit to Stefan-1313/ray_mod that referenced this pull request Aug 18, 2022
@krfricke
[air] Serialize additional files in dict checkpoints turned dir check…
93ba4a6 
…points (ray-project#26351)

With this PR, files put into directory checkpoints that were dict checkpoints will be serialized and retained when a subsequent to_dict() is called. This is to enable storing additional files, as e.g. needed by Ray Tune.

Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Stefan van der Kleij <[email protected]>
Stefan-1313 pushed a commit to Stefan-1313/ray_mod that referenced this pull request Aug 18, 2022
@krfricke
[tune] Use Checkpoint.to_bytes() for store_to_object (ray-project#2…
81a1c53 
…5805)

We currently use our own serialization to ship checkpoints as objects. Instead we should use the Checkpoint class. This PR also adds support to create results from checkpoints pointing to object references.

Depends on ray-project#26351

Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Stefan van der Kleij <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Yard1 Yard1 Yard1 approved these changes

@xwjiang2010 xwjiang2010 xwjiang2010 approved these changes

Assignees