DeepSpeed ZeRO++

lastmod 2023-06-28created 2023-06-281 edits

A leap in LLM and chat model training speed with 4x less communication

Features

• LLMs require quite a large amount of memory and compute resources
• DeppSpeed's ZeRO family offers a solution to this problem and is used in TNLG-17B, Bloom-176B, MPT-7B, Jurrasic-1, and others
• However, in large-scale scenarios with many GPUs, overhead arises, such as the need for frequent communication between GPUs
• For such cases, ZeRO++ reduces total communication volume by 4x without affecting model quality
  • Accelerates large-model pre-training and fine-tuning
    • Small batch size per GPU: 2.2x the throughput of ZeRO
    • Even on low-bandwidth clusters, performs similarly to 4x the bandwidth
  • Also accelerates ChatGPT-like models that use RLHF

See also

• LLM

Favorite site

• DeepSpeed ZeRO++: A leap in speed for LLM and chat model training with 4X less communication - Microsoft Research