Megabyte

lastmod 2023-05-30created 2023-05-262 edits

Meta unveils Megabyte, an innovative AI model architecture

Features

• Goes beyond the transformer architecture used by models like GTP-4, scaling to more efficient/parallel processing
• Transformers suit short sequences, but are hard to scale to long inputs of over 1 million tokens, such as high-resolution images, podcasts, code, and books
• Megabyte is a multi-scale decoder architecture that can model sequences of over 1 million
  • Splits input and output sequences into "Patches" rather than individual tokens
  • A local AI model generates output for each patch, and a global model manages and coordinates these patches
• In tests, a Megabyte model with 1.5 billion (1.5B) parameters built sequences 40% faster than a transformer model running on 350 million (350M) parameters
• In tests, it handled up to 1.2M tokens, far beyond GPT-4's 32,000 tokens and Claude's 100,000 tokens

Meta AI's Megabyte explained

• An explanation of Meta AI's revolutionary model architecture MEGABYTE (MEGABYTE, Meta AI’s New Revolutionary Model Architecture, Explained) - Reading & Info Sharing - PyTorch Korea User Group

A translation of the Megabyte explainer written by ENCORD, a startup building AI infrastructure / tools

• Introduces the main components of MegaByte
  • Patch Embedder: embeds the input and splits it into patches
  • Global Module: an autoregressive transformer that performs self-attention
  • Local Module: predicts the next patch from the input received from the global model
• Introduces the Multiscale Transformer
• Introduces the Autoregressive Transformer
• Problems with current models: Tokenization, Scalability, Generation Speed
• Proposed solutions
  • Self-attention with computational cost reduced to O(N^(4/3))
  • Feedforward layers applied per patch
  • Parallel processing during decoding
• (Addendum) Recent news from Meta AI
  • SAM: https://news.hada.io/topic?id=8893
  • MTIA: https://news.hada.io/topic?id=9246
  • DINOv2: https://news.hada.io/topic?id=9269
  • ImageBIND: https://news.hada.io/topic?id=9156
  • (Not in the article, but) MMS: https://news.hada.io/topic?id=9245
  • (Also not in the article, but) LIMA: https://news.hada.io/topic?id=9239

See also

• Deep learning
• Meta

Favorite site

• Meta AI Unleashes Megabyte, a Revolutionary Scalable Model Architecture - Artisana