Transformer

lastmod 2026-03-08created 2023-04-2513 edits

The Transformer is a model from the paper "Attention is all you need", published by Google in 2017. It follows the encoder-decoder structure of existing seq2seq models, but, as the paper's title says, is implemented with Attention alone. Although this model designs an encoder-decoder structure without using RNNs, it showed better translation performance than RNNs.

Categories

• Self Attention

Projects

• Bark - Korean: 트랜스포머-based Text-To-Audio model
• BitNet-Transformers - Weights made of Integer values (faster training, smaller model size) etc .. <- needs checking.
• RustGPT - pure Transformer LLM implemented entirely from scratch in Rust
• ntransformer - NVMe-to-GPU inference engine that runs Llama 3.1 70B on a single RTX 3090

Attention

• Attention in transformers, step-by-step | DL6 - YouTube - 3Blue1Brown

Documentation

How the Transformer Works v1 (b5t)

See also

• Deep learning
• TTS
• seq2seq
• NLP
• HuggingFace:Transformers
• MiniMax-M1 - open-weight, large-scale hybrid-attention reasoning model

Favorite site

• Transformer (deep learning architecture) - Wikipedia
• 16-01 Transformer - Introduction to Natural Language Processing with Deep Learning
• The Transformer explained with illustrations | GeekNews
  • [Recommended] [Original] The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time. ¹

References