Falcon

lastmod 2023-09-08created 2023-09-081 edits

180B model released (the largest open language model, with 180 billion parameters)

Features

• The largest open language model, with 180 billion parameters
• Ranked #1 on the leaderboard among open models. Outperforms Llama 2 70B and GPT-3.5 and competes with PaLM-2
• Trained on 3.5T tokens using TII's RefinedWeb dataset (mostly English)
  • 2.5 times larger than Llama 2 and trained with more than 4 times the compute (using 4096 GPUs on Amazon SageMaker)
• Falcon 180B can be used commercially, but only under very restrictive conditions, excluding "hosting use". Be sure to check the license
• Hardware requirements
  • Full fine-tuning: 5120GB memory, 8x 8x A100 80GB
  • LoRA with ZeRO-3: 1280GB, 2x 8x A100 80GB
  • QLoRA: 160GB, 2x A100 80GB
  • Inference BF16/FP16 : 640GB, 8x A100 80GB
  • Inference GPTQ/int4 : 320GB, 8x A100 40GB

See also

• LLM

Favorite site

• Spread Your Wings: Falcon 180B is here