Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Add Lightning 2.0 compatibility test pipeline #34147

Merged
merged 24 commits into from
Apr 27, 2023
Merged

[CI] Add Lightning 2.0 compatibility test pipeline #34147

merged 24 commits into from
Apr 27, 2023

Conversation

woshiyyya
Copy link
Member

@woshiyyya woshiyyya commented Apr 6, 2023 •
edited
Loading

Why are these changes needed?

PyTorch Lightning recently updated to version 2.0, but our CI pipelines only covered Lightning 1.x. This pull request aims to build a new pipeline to test the compatibility of LightningTrainer with PTL v2.0.

Some minor changes to unit tests:

  • Updated the testing LightningModule (remove the deprecated validation_epoch_end())
  • Use different syntax to feed model params to optimizer when using FSDP.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@woshiyyya
init
3a678ec 
Signed-off-by: woshiyyya <[email protected]>
@woshiyyya
fix buildkite
bab9887 
Signed-off-by: woshiyyya <[email protected]>
@woshiyyya
fix pipeline tag
7a5910b 
Signed-off-by: woshiyyya <[email protected]>
@woshiyyya
change test model definition for ptl20
6d0668d 
Signed-off-by: woshiyyya <[email protected]>
@woshiyyya
specify py38
723f934 
Signed-off-by: woshiyyya <[email protected]>
@woshiyyya
fixing
976a9a1 
Signed-off-by: woshiyyya <[email protected]>
@woshiyyya
Merge remote-tracking branch 'upstream/master' into add_py38_compat_p…
d8c83da 
…ipeline
@woshiyyya
fixing
f36ec3d 
Signed-off-by: woshiyyya <[email protected]>
@woshiyyya woshiyyya changed the title [CI] Add py38 compatibility pipeline [CI] Add Lightning 2.0 compatibility test pipeline Apr 13, 2023
woshiyyya and others added 10 commits April 12, 2023 17:51
@woshiyyya woshiyyya marked this pull request as ready for review April 20, 2023 07:16
justinvyu
justinvyu reviewed Apr 25, 2023
View reviewed changes
Copy link
Contributor

@justinvyu justinvyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Left some questions.

.buildkite/pipeline.gpu_large.yml Outdated Show resolved Hide resolved
.buildkite/pipeline.gpu_large.yml Show resolved Hide resolved
.buildkite/pipeline.gpu_large.yml Show resolved Hide resolved
ci/env/install-minimal.sh Show resolved Hide resolved
python/ray/train/BUILD Show resolved Hide resolved
python/ray/train/tests/lightning_test_utils.py Show resolved Hide resolved
justinvyu
justinvyu approved these changes Apr 26, 2023
View reviewed changes
Copy link
Contributor

@justinvyu justinvyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the clarifications.

python/ray/train/tests/lightning_test_utils.py Show resolved Hide resolved
krfricke
krfricke approved these changes Apr 27, 2023
View reviewed changes
Copy link
Contributor

@krfricke krfricke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@krfricke krfricke merged commit aeed2b3 into ray-project:master Apr 27, 2023
scv119 added a commit that referenced this pull request May 1, 2023
@scv119
Revert "[CI] Add Lightning 2.0 compatibility test pipeline (#34147)"
ad93b6b 
This reverts commit aeed2b3.
@scv119 scv119 mentioned this pull request May 1, 2023
Merged
scv119 added a commit that referenced this pull request May 2, 2023
@scv119
Revert "[CI] Add Lightning 2.0 compatibility test pipeline (#34147)" (#…
24c3c4f 
…34933)

This reverts commit aeed2b3.
woshiyyya added a commit that referenced this pull request May 2, 2023
@woshiyyya
Revert "Revert "[CI] Add Lightning 2.0 compatibility test pipeline (#…
a4d0c86 
…34147)" (#34933)"

This reverts commit 24c3c4f.
This was referenced May 2, 2023
Merged
Open
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@woshiyyya @ProjectsByJackHe
[CI] Add Lightning 2.0 compatibility test pipeline (ray-project#34147)
e67bff1 
PyTorch Lightning recently updated to version 2.0, but our CI pipelines only covered Lightning 1.x. This pull request aims to build a new pipeline to test the compatibility of LightningTrainer with PTL v2.0.

Some minor changes to unit tests:
- Updated the testing LightningModule (remove the deprecated `validation_epoch_end()`)
- Use different syntax to feed model params to optimizer when using FSDP.

Signed-off-by: woshiyyya <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Co-authored-by: Kai Fricke <[email protected]>
Signed-off-by: Jack He <[email protected]>
architkulkarni pushed a commit to architkulkarni/ray that referenced this pull request May 16, 2023
@woshiyyya @architkulkarni
[CI] Add Lightning 2.0 compatibility test pipeline (ray-project#34147)
742b5c6 
PyTorch Lightning recently updated to version 2.0, but our CI pipelines only covered Lightning 1.x. This pull request aims to build a new pipeline to test the compatibility of LightningTrainer with PTL v2.0.

Some minor changes to unit tests:
- Updated the testing LightningModule (remove the deprecated `validation_epoch_end()`)
- Use different syntax to feed model params to optimizer when using FSDP.

Signed-off-by: woshiyyya <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Co-authored-by: Kai Fricke <[email protected]>
architkulkarni pushed a commit to architkulkarni/ray that referenced this pull request May 16, 2023
@scv119 @architkulkarni
Revert "[CI] Add Lightning 2.0 compatibility test pipeline (ray-proje…
9cf0ebd 
…ct#34147)" (ray-project#34933)

This reverts commit aeed2b3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@justinvyu justinvyu justinvyu approved these changes

@krfricke krfricke krfricke approved these changes

@matthewdeng matthewdeng Awaiting requested review from matthewdeng matthewdeng is a code owner

Assignees

@krfricke krfricke

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@woshiyyya @justinvyu @krfricke