Discussions

• 2 You must be logged in to vote

  🙏

  Global_steps taking too much space

  pratikkumar018 asked Sep 12, 2023 in Q&A · Answered

  

  

  

  

  5
• 3 You must be logged in to vote

  💬

  Is deepspeed zero stage 3 MP or PP. Same as FSDP?

  andrasiani started May 6, 2023 in General

  

  

  

  

  6
• 1 You must be logged in to vote

  🙏

  When using torch.no_grad(), will deepspeed *not* perform memory optimization on some parameters of no_grad?

  ojipadeson asked Jul 29, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  How to set specified stage to specified device in pipeline parallelism?

  ShuaipengWu asked Jul 11, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Can deepspeed's zero_optimization achieve model parallelism?

  ojipadeson asked Jul 1, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  How to set a layer like LayerNorm to fp32 while fp16 is enabled?

  forrestjgq asked Jun 18, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  The resume from checkpoint seems not working with Zero3-offload on NVME.

  bug-fixed asked May 30, 2024 in Q&A · Unanswered

  

  

  1
• 2 You must be logged in to vote

  💬

  PyTorch memory profiler memory timeline not showing categories when used with DeepSpeed Zero-3

  drwslacy47 started May 30, 2024 in General

  

  

  0
• 8 You must be logged in to vote

  💬

  Can't get Deepspeed to work on Ubuntu 22.04

  AlpsAficionado started May 12, 2023 in General

  

  

  2
• 2 You must be logged in to vote

  🙏

  Pipeline parallelism for models in transformers

  KaiLv69 asked Apr 23, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Why ZeRO-offload parameter update after model backward? can do them pipeline?

  Zijie-Tian asked Apr 29, 2024 in Q&A · Answered

  

  

  2
• 2 You must be logged in to vote

  🙏

  Gradient synchronization in ZeRO-1/2/3

  R0n12 asked May 22, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Problem Unable to pre-compile async_io while pip

  YolandeJ asked May 23, 2024 in Q&A · Unanswered

  

  

  0
• 3 You must be logged in to vote

  💬

  Does the distributed KQ^T in DeepSpeed Ulysses make sense?

  brunomaga started May 20, 2024 in General · Closed

  

  

  2
• 1 You must be logged in to vote

  🙏

  Saving checkpoints in deepspeed

  base-y asked May 1, 2022 in Q&A · Unanswered

  

  

  

  

  5
• 1 You must be logged in to vote

  🙏

  zero1 + overlap_comm : Is this correct?

  Suparjie asked May 9, 2024 in Q&A · Unanswered

  

  

  1
• 2 You must be logged in to vote

  🙏

  Inquiry Regarding DeepSpeed Trademark Use for Hardware Vendor Fork

  question Further information is requested legal Questions and discussions related to legal matters.

  kevint324 asked Apr 11, 2024 in Q&A · Closed · Answered

  

  

  3
• 1 You must be logged in to vote

  🙏

  Poor GPU utilization when using multi-node ZeRO-3 on small model

  BramVanroy asked Apr 30, 2024 in Q&A · Closed · Answered

  

  

  7
• 2 You must be logged in to vote

  💬

  Invalidate trace cache @ step 13: expected module xx, but got module xxx

  mrluin started Aug 3, 2023 in General

  

  

  

  

  12
• 1 You must be logged in to vote

  🙏

  How to use TiedLayerSpec

  josephwong14wkh asked Apr 29, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  AutoTP for training?

  KaiLv69 started Jan 11, 2024 in General

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  How to set different learning rate for different type of weights?

  tusiqi1 asked Apr 22, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Why activations memory is computed through an experiment rather formulating it for DeepSpeed autotuner?

  kmehant asked Apr 19, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Custom tensor parallelism

  sarathkondeti asked Apr 19, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  LMOD: Error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory

  BramVanroy asked Jan 20, 2023 in Q&A · Answered

  

  

  

  

  6