MiniGPT-4
lastmod 2023-04-18created 2023-04-181 edits
Enhancing vision-language understanding with advanced LLMs
Features
- Provides new vision-language capabilities similar to those demonstrated in GPT-4
- Describes images in detail, explains how to cook a dish from a food photo, finds problems, writes ad copy, and writes stories and poems inspired by images
- Trained on 5 million image-text pairs with 4 A100s for 10 hours. After this stage alone Vicuna understands images, but its generation ability is heavily affected
- To solve the problem and improve usability, proposes a new method of generating high-quality image-text pairs using the model itself together with ChatGPT
- Based on this, creates a small (3,500 pairs in total) high-quality dataset
- In the second fine-tuning stage, trains on this small dataset with a conversational template to improve generation reliability and overall usability
- Surprisingly, this stage is computationally efficient and takes only 7 minutes on a single A100