RedPajama

lastmod 2023-05-10created 2023-04-193 edits

An open-source project to recreate the LLaMA dataset

Features

• A project to build reproducible, fully open language models, rather than semi-open models like LLaMA, Alpaca, and Vicuna
• 3 components
  • Pre-Training data with high quality and broad coverage
  • Base models trained at scale on this data
  • Instruction tuning data and models to make the base models safe and usable
• Released the RedPajama-Data-1T dataset as the first component
  • A fully open dataset of 1.2 trillion tokens, created by following the recipe described in the LLaMA paper
  • Downloadable via HuggingFace. 5TB in total (distributed compressed to 3TB)
  • Made up of 7 data slices: each preprocessed and filtered to roughly match the counts in the LLaMA paper (the preprocessing methods and filters are also published on GitHub)
    • CommonCrawl (878b) - web crawl data
    • C4 (175b) - Colossal, Cleaned version of Common Crawl
    • GitHub (59b) - GitHub data filtered by license and quality
    • arXiv (28b) - scientific papers and articles (boilerplate removed)
    • Books (26b) - corpus of public books, deduplicated by content similarity
    • Wikipedia (24b) - a subset of Wikipedia pages (boilerplate removed)
    • StackExchange (20b) - a subset of StackExchange pages (boilerplate removed)
• The next step is to train a strong base model. To be released within a few weeks
• Instruction tuning is planned to use what is provided through OpenChatkit

See also

• LLaMA
• OpenChatKit released - an open-source project for building something like ChatGPT

Favorite site

• RedPajama, a project to create leading open-source models, starts by reproducing LLaMA training dataset of over 1.2 trillion tokens — TOGETHER
• RedPajama 3B, 7B models released | GeekNews