Attention
lastmod 2026-03-08created 2026-03-081 edits
In neural networks, attention is a technique devised to mimic attention in cognitive psychology. Attention enhances the effect of some parts of the input data while diminishing other parts. This is so that the network focuses more on the small but important parts of the data. Which part of the data is more important to learn than another depends on the context, and this is trained by gradient descent.
Attention and similar mechanisms were introduced in the 1990s under names such as multiplicative modules, sigma pi units, and hypernetworks. The flexibility of attention comes from its role as "soft weights" that can change during runtime, unlike standard weights that remain fixed during runtime.
Attention is used in memory in neural Turing machines, reasoning tasks in differentiable neural computers, language processing in transformer-family models, and multimodal data processing (sound, images, video, text) in perceivers.