Vision-language-action model

lastmod 2026-05-29created 2026-01-1511 edits

A VLA model (Vision-Language-Action Model) refers to robot foundation models that take inputs such as text, video, and demonstrations and generate actions. In other words, it is a kind of generative AI that goes into AI robots.

Keywords

• Humanoids that learn from video
  • Physical-world adaptation and on-device acceleration of VLA models
• Robot actions that adapt to changing environments (Adapting actions to changing environment)
• Development and commercialization of VLA-based on-device AI in manufacturing sites
  • Strengthening perception and control in industrial robot automation systems
  • AI model lightweighting and optimization
  • VLA-based video surveillance solutions
• Latest trends and use cases of VLM/VLA models for robot action learning
  • High-level Reasoning vs Low-level Control (the speed imbalance problem in VLA architectures)
• Physical AI toward Zero-shot robot tasks: strategies for applying it in manufacturing sites
  • VLA overfitting and data bottleneck problems
  • Physical Skill Foundation: Sim-to-Real and Foundation Model-based approaches
• Applying VLA models to humanoids
  • Cases and discussion of imitation learning-based object manipulation tasks
  • Sharing cases of applying VLA to humanoid hands
• Multi-stage VLA architecture and Physical AI implementation techniques for industrial sites
  • Multi-stage VLA architecture for processing multimodal input
  • Task planning and robot action execution techniques for industrial sites
  • 2D and 3D vision recognition techniques for understanding the work environment

Main structure and components

• 🧠 Vision-Language-Action (VLA)

Vision Encoder

• Converts images or video frames into high-dimensional representations
• Uses ViT, CLIP, ResNet-based VLMs

Language Encoder

• Converts text and voice commands into semantic embeddings
• Uses GPT, LLaMA, Gemini 1.5, LLM-based VLMs (AI usage info sharing, The Miilk)

High-Level Policy

• Feeds the VLM visual + language input together with a demo tour video (long context) to predict the goal frame (goal)
• Plays a key role in the MINT (Multimodal Instruction Navigation with Tours) task (Moonlight)

Low-Level Policy

• Uses a topological graph automatically generated via COLMAP
• Plans the shortest path with Dijkstra's method and executes actions (wp) in the form Δx, Δy, Δθ (Moonlight)

Overall pipeline

[멀티모달 입력(텍스트·이미지·음성) + 시연 비디오] 
→ 상위 정책(VLM 처리) → 목표 프레임 인식 
→ 하위 정책(위상 그래프 기반 경로 생성) 
→ 물리적 이동/조작 수행

Representative models and latest research trends

• Mobility VLA (Google DeepMind)
  • Achieved an 86–90% success rate in a real-world 836m² office (AI usage info sharing)
  • Based on long context (long-context VLM + Gemini 1.5 Pro); can carry out reasoning-driven multimodal commands (The Miilk)
• RT-2 (Google DeepMind)
  • A representative realization of VLA; Vision + Language → maps directly to actions (Wikipedia)
• OpenVLA (Stanford et al.)
  • 7B parameter open-source VLA based on 970k robot demonstrations
  • 16.5% higher success rate than RT‑2‑X; supports different robot architectures (arXiv)
• TinyVLA
  • Lightweight VLA model with fast inference and data efficiency
  • Distributed/diffusion (decoder+a diffusion policy) structure, extending to real-world control (arXiv)
• OTTER
  • Selects only the visual features that match the text instruction, so it can execute instructions directly while keeping the VLM frozen
• Helix (Figure AI)
  • The first case to support humanoid upper-body and finger control, and even multi-robot collaboration

Technical issues and solution strategies

• Multimodal alignment & long context
  • Achieves complex natural language reasoning by fusing long demonstration videos + VLM (The Miilk, Moonlight)
• Hierarchical policy structure
  • Splits roles into upper: goal recognition, lower: path control → maximizes efficiency and accuracy
• Real-world deployment problem (Sim2Real)
  • Path planning robust to change by using topological graphs and COLMAP
• Control frequency and action representation
  • Research underway for fast action generation, such as π0's FAST and TinyVLA's diffusion decoder (AI usage info sharing, learnopencv.com)

Applications

• Robotic Navigation
• Service/home/logistics robots
• Everyday humanoid control
• Action commands in industrial automation and autonomous driving
• HRI (Human–Robot Interaction), remote control and education

Summary comparison table

Documentation

https://arxiv.org/abs/2505.04769

[2505.04769] Vision-Language-Action (VLA) Models - Concepts, Progress, Applications and Challenges

See also

• OpenVLA
• AI
• Deep learning
• Multimodal learning
• LLM

Favorite site

• Vision-language-action model - Wikipedia
• VLA - Namuwiki
• Vision-Language-Action Models: Concepts, Progress, Applications and Challenges -- (1)
• 🧠 Vision-Language-Action (VLA)