FastVLM
lastmod 2025-05-14created 2025-05-141 edits
This repository contains the official implementation of "FastVLM: Efficient Vision Encoding for Vision Language Models" - CVPR 2025
Efficient vision encoding for vision language models
About
- The official Repo for "FastVLM: Efficient Vision Encoding for Vision Language Models", presented by Apple at CVPR 2025
- FastViTHD reduces the number of tokens and shortens encoding time for high-resolution images
- The smallest model achieves results 85x faster than LLaVA-OneVision-0.5B with a 3.4x smaller encoder
- The large model outperforms Cambrian-1-8B and is 7.9x faster
- A demo app that runs on mobile devices such as the iPhone is provided
Significance and advantages of the FastVLM project
- FastVLM is an open-source official implementation for vision language models (Vision Language Model, VLM)
- Offers clear advantages in speed and efficiency over existing vision encoders
- Highly useful across various hardware, especially Apple Silicon and mobile environments
- You can pick pretrained models of various sizes and performance levels yourself
- Compared to other projects, its smaller model sizes ensure optimized real-time responses and low hardware resource usage
Key features
- FastViTHD is an innovative hybrid vision encoder that reduces the number of output tokens, greatly shortening high-resolution image encoding time
- The smallest FastVLM-0.5B model has an 85x faster TTFT (time to first token) and a 3.4x smaller encoder than LLaVA-OneVision-0.5B
- The large FastVLM-7B model combined with the Qwen2-7B LLM shows strong performance with a single image encoder and 7.9x faster TTFT compared to recent SOTA such as Cambrian-1-8B
- A demo app that runs in a real mobile environment (iOS) is also provided, so you can verify the technology's usefulness right away
Model information (Model Zoo)
- FastVLM models of various sizes (FastVLM-0.5B, FastVLM-1.5B, FastVLM-7B) are provided in stage 2 and stage 3 versions
- PyTorch checkpoint files are officially provided for each model
- Users can batch-download multiple models into the checkpoints directory with the officially provided command
Usage Example
- You can quickly and easily test inference with a pretrained PyTorch checkpoint using the predict.py script
- With the example command, you input an image and give a prompt (question) to get a description of the image or an answer to the question
Apple Silicon and mobile device support
- A guide is provided that explains the separate model export and quantization process for inference on Apple Silicon
- Checkpoint files optimized directly for Apple Silicon are officially distributed
- An app development guide and source code that can be used right away on iPhone, iPad, Mac, etc. are in the /app folder
Additional information and open-source notes
- The official arXiv link for the FastVLM paper and the CVPR 2025 conference paper citation format are provided
- The codebase builds on various open-source projects, and contributions and license information are described separately
- Be sure to check the licenses (the license file and model license) before using the models and code