WIP: Disable _optimizer_to_device logic

[corwinjoy]

This PR exists to test what happens when we disable _optimizer_to_device.

Pursuant to discussion in #19955

Before submitting

• Was this discussed/agreed via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

---

📚 Documentation preview 📚: https://pytorch-lightning--20036.org.readthedocs.build/en/20036/

[corwinjoy]

Running the tests locally this causes the following tests to fail:

> FAILED tests/tests_pytorch/loops/test_evaluation_loop_flow.py::test__eval_step__flow - RuntimeError: Expected all tensors to be on the same device, but found at l...
6712a7097
> FAILED tests/tests_pytorch/loops/test_training_loop_flow_scalar.py::test__training_step__epoch_end__flow_scalar - RuntimeError: Expected all tensors to be on the same device, but found at l...
6713a7099
> FAILED tests/tests_pytorch/trainer/test_trainer.py::test_multiple_trainer_constant_memory_allocated - AssertionError: assert device(type='cuda', index=0) == device(type='cpu')
6824c7210

In addition I would expect tests/tests_fabric/utilities/test_optimizer.py to fail where it checks all params are moved to the device. This test does not seem to run by default but I think we can agree that this test is incorrect since we want only some parameters moved to the device depending on the optimizer.

[awaelchli]

Thanks for checking this @corwinjoy. I'll trigger all tests now in the CI because locally not all of them may run.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 57%. Comparing base (37e04d0) to head (d2589fc).
Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (37e04d0) and HEAD (d2589fc). Click for more details.

HEAD has 230 uploads less than BASE

Flag	BASE (37e04d0)	HEAD (d2589fc)
examples	6	0
cpu	91	36
pytest	59	21
app	6	0
gpu	4	3
lightning_fabric	13	7
lightning	58	26
lightning_app	5	0
pytorch_lightning	13	6
python3.11	28	14
python3.8	24	12
python3.10	22	10
pytorch2.1	24	12
pytest-full	36	18
pytorch2.3	6	3
pytorch2.2	6	3

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #20036     +/-   ##
=========================================
- Coverage      83%      57%    -26%     
=========================================
  Files         427      422      -5     
  Lines       35460    35358    -102     
=========================================
- Hits        29599    20167   -9432     
- Misses       5861    15191   +9330