XGen-7B

lastmod 2023-07-01created 2023-07-011 edits

• XGen-7B - A 7B LLM trained on 1.5T tokens with up to 8K sequence length

Features

• As LLMs become widely used, applying them to long sequences has become important: document summarization, code writing, protein sequence prediction, etc.
• However, most open-source LLMs (LLaMA, MPT, Falcon) are trained with a maximum sequence length of 2K tokens
• XGen-7B is trained on 1.5T tokens with sequence lengths of up to 8K
• On standard NLP benchmarks, performance on par with or better than MPT, Falcon, LLaMA, Redpajama, and OpenLLaMA of the same model size
• Strong results on both text (MMLU, QA) and code (HumanEval) tasks
• Training cost of about $150K for 1T tokens on TPU-v4

See also

• LLM

Favorite site

• Long Sequence Modeling with XGen: A 7B LLM Trained on 8K Input Sequence Length