GPT-Neo
lastmod 2021-01-19created 2021-01-192 edits
GPT-Neo is the code name for a series of transformer-based language models loosely styled around the GPT architecture that we plan to train and open source. Our primary goal is to replicate a GPT-3 sized model and open source it to the public, for free.
Along the way we will be running experiments with alternative architectures and attention types, releasing any intermediate models, and writing up any findings on our blog.
We have a codebase built in Tensorflow-mesh (for training on TPUs), and one built with Deepspeed (for training on GPUs). Both can scale to GPT-3+ sizes, but we currently lack the TPUs to train a 175B model to completion. Thankfully, we don't lack GPUs.
GPT-Neo is now fairly stable, and we will be releasing smaller scale models shortly. GPT-Neox is still a work in progress, and we will be releasing more updates as the project moves forward.
Features
- Developing two implementations that can scale to GPT-3+ sizes
- GPT-Neo : code based on Tensorflow-mesh (TPU)
- GPT-Neox : code based on DeepSpeed (GPU)