Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle empty parameter groups #4277

Merged
merged 3 commits into from
Sep 13, 2023
Merged

Handle empty parameter groups #4277

merged 3 commits into from
Sep 13, 2023

Conversation

tjruwase
Copy link
Contributor

@tjruwase tjruwase commented Sep 6, 2023

Filter out empty parameter groups in both engine and zero3.

Fix #2820

@awan-10
Copy link
Contributor

awan-10 commented Sep 6, 2023

This PR is the proper fix and is linked to #4270. But both fixes will only work with HF transformers==4.31.0. New HF versions are still broken with zero-stage 3. Actively being debugged by @tjruwase and @awan-10 .

@tjruwase tjruwase added this pull request to the merge queue Sep 12, 2023
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 12, 2023
@tjruwase tjruwase added this pull request to the merge queue Sep 13, 2023
Merged via the queue into master with commit 9adc73f Sep 13, 2023
16 checks passed
CurryRice233 pushed a commit to CurryRice233/DeepSpeed that referenced this pull request Sep 15, 2023
* origin/master: (48 commits)
  Fix autotune to support Triton 2.1  (microsoft#4340)
  Fix skipped inference tests (microsoft#4336)
  Suppress noise (microsoft#4310)
  Fix a bug in the implementation of dequantization for inference (microsoft#3433)
  DS-Chat BLOOM: Fix Attention mask (microsoft#4338)
  clear redundant timers (microsoft#4308)
  Add release version checking (microsoft#4328)
  Fix Zero3 contiguous grads, reduce scatter false  accuracy issue (microsoft#4321)
  Clean up modeling code (microsoft#4320)
  Handle empty parameter groups (microsoft#4277)
  Update README.md (microsoft#4316)
  README update (microsoft#4303)
  Update release and bump patch versioning flow (microsoft#4286)
  added a bert-model check for triton (microsoft#4266)
  ZeRO-Inference v2 release
  bump to 0.10.4
  Update index.md (microsoft#4297)
  fix user args parsing of string with spaces on runner (microsoft#4265)
  ZeRO-Inference refresh (microsoft#4197)
  AMD Kernel Compatibility Fixes (microsoft#3180)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] ValueError: max() arg is an empty sequence using bf16 zero stage3
2 participants