LoRA
lastmod 2025-02-03created 2023-03-048 edits
(For the wireless communication technology for sensor networks, see LoRa (lowercase 'a'))
LoRA: Low-Rank Adaptation of Large Language Models
Summary
LoRa is one of the fine-tuning techniques for Stable Diffusion.
By reducing the parameters used in training, it makes fine-tuning faster and more effective. (Reportedly it can train even with 7GB of VRAM.)
Like Dreambooth, you can input a few images and generate pictures in that style.
LoRa stands for Low-Rank Adaptation. Low rank adaptation is a mathematical technique that can reduce the parameters used in training. Think of it as an effective way to create the difference (diff) between models instead of storing the whole model. Lora was developed by researchers at Microsoft and applied to stable diffusion by Simo Ryu.
Abstract
Proposes Low-Rank Adaptation (LoRA), which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the transformer architecture, greatly reducing the number of parameters for downstream tasks.
Introduction
LoRA was inspired by Measuring the Intrinsic Dimension of Objective Landscapes.
The researchers hypothesized that the change in weights during model adaptation has a low intrinsic rank, and proposed LoRA (Low-Rank Adaptation).
Documentation
- [2106.09685] LoRA - Low-Rank Adaptation of Large Language Models
- https://arxiv.org/abs/2106.09685
See also
- Stable Diffusion
- Textual Inversion
- Dreambooth
- LoRA
- LoRa (in the IoT field) - (lowercase 'a')
Favorite site
- Github - LoRA
- LoRA: Low-Rank Adaptation of Large Language Models
- How to create realistic photos with AI. So easy and simple you can do it right away. - YouTube
- How to Fine-tune Stable Diffusion using LoRA | by Ng Wai Foong | Feb, 2023 | Medium
- Generating multiple LoRA models combined into a single image | GeekNews