Fix GradScaler import on torch >= 2.3.0 #620
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #620 +/- ##
==========================================
+ Coverage 85.39% 86.07% +0.67%
==========================================
Files 81 81
Lines 8006 8014 +8
==========================================
+ Hits 6837 6898 +61
+ Misses 1169 1116 -53 |
benchmarks/benchmark_optimizer.py
Outdated
| @@ -98,7 +98,7 @@ def run_trainer(batch_size: int, batch_time: float, client_mode: bool, verbose: | |||
| grad_scaler = hivemind.GradScaler() | |||
| else: | |||
| # check that hivemind.Optimizer supports regular PyTorch grad scaler as well | |||
| grad_scaler = torch.cuda.amp.GradScaler(enabled=args.use_amp) | |||
| grad_scaler = torch.amp.GradScaler(enabled=args.use_amp) | |||
There was a problem hiding this comment.
One small question just to make sure: if the user has torch==1.9.0 (earliest supported version in requirements.txt), does this version already have torch.amp? Maybe we need to bump the version in requirements to make sure it won't break existing code?
There was a problem hiding this comment.
This part used to exist, but got lost somewhere in splitting all of this work into multiple PRs. I've restored the logic that imports this GradScalar differently here, depending on the installed Torch version.
To answer your question, no, torch.amp does not exist in 1.9.
No description provided.