Large Multimodal Model

lastmod 2026-04-26created 2024-07-027 edits

A multimodal model is an artificial intelligence that considers various types of data (modalities) together, such as text, images, audio, and video, and learns and processes the relationships among them. Among these, relatively large models are called "giant multimodal models" or "large multimodal models (LMM (Large Multimodal Model))".

Types

• Multimodal model - Namuwiki

Early Fusion

Early Fusion first combines two different kinds of data into a single piece of data and then trains the model. Various data transformations are performed to combine the two differently formatted data. The raw data may be fused as is, or fused after preprocessing.

Late Fusion

Late Fusion trains two different kinds of data on separate models and then fuses the resulting outputs, similar to how conventional ensemble models work.

Joint Fusion

Joint Fusion does not train the two modalities' data simultaneously and has the flexibility to merge modalities at whatever model depth you want. Model training proceeds with one modality and fuses it with another modality before the last layer of training; this process is also called end-to-end learning.

Categories

• LLaVA
• EAGLE - NVIDIA releases as open source a new multimodal language model (LMM) that greatly improves the ability to understand and interact with complex visual information
• Llama 3.2 - Revolutionizing edge AI and vision with open, customizable models (Meta released the first large multimodal model (LMM) in its 'Llama' series that understands both images and text)
• Magma - a foundation model for multimodal AI agents

See also

• LLM
• Multimodal learning
• Vision language action model

Favorite site

• A powerful being that has learned even images: LMMs are coming!
• LLM vs LMM: The language model of the future? · Blog - datamaker
• (IT basics) Definitions of large language models (LLM) and large multimodal models (LMM), and GPT-4V : Naver Blog
• Arxiv Dives - LLaVA 🌋 an open source Large Multimodal Model (LMM)
• Multimodality and Large Multimodal Models (LMMs)
• The dawn of the LMM (Large Multimodal model)s era, seen through GPT-4V - Part 1
• BradyFU/Awesome-Multimodal-Large-Language-Models: :sparkles::sparkles:Latest Advances on Multimodal Large Language Models