Vision Langunage Model
lastmod 2026-07-28created 2026-07-272 edits
A vision language model (VLM) is an artificial intelligence (AI) model that combines computer vision and natural language processing (NLP) capabilities.
About
VLMs learn to map the relationships between text data and visual data such as images or videos, so they can generate text from visual input or understand natural language prompts in the context of visual information.
Also called visual language models, VLMs combine a large language model (LLM) with a vision model or visual machine learning (ML) algorithms.
As multimodal AI systems, VLMs take text and images or videos as input and output text, usually in the form of image or video descriptions, to answer questions about an image or to identify parts of an image or objects in a video.
Elements of a vision language model
A vision language model generally consists of two main components:
- Language encoder
- Vision encoder
Language encoder
The language encoder captures the semantic meaning and contextual associations between words and phrases and converts them into text embeddings that an AI model can process.
Most VLMs use a neural network architecture known as the transformer model for the language encoder. Examples of transformers include Google's BERT (Bidirectional Encoder Representations from Transformers), one of the first foundation models and the basis of many of today's LLMs, and OpenAI's GPT (Generative Pre-trained Transformer).
Here is a brief overview of the transformer architecture:
- The encoder converts input sequences into numerical representations called embeddings, which capture the meaning and position of tokens in the input sequence.
- The self-attention mechanism lets the transformer "focus its attention" on the most important tokens in the input sequence, regardless of their position.
- The decoder uses this self-attention mechanism and the encoder's embeddings to generate the statistically most probable output sequence.
Vision encoder
The vision encoder extracts important visual properties such as color, shape, and texture from image or video input and converts them into vector embeddings that a machine learning model can process.
Earlier versions of VLMs used deep learning algorithms such as convolutional neural networks for feature extraction. Modern vision language models use a vision transformer (ViT), which applies elements of transformer-based language models.
A ViT processes an image as patches and treats them as a sequence, like tokens in a language transformer. The vision transformer then implements self-attention over these patches to create a transformer-based representation of the input image.