Stable Diffusion
lastmod 2026-02-26created 2022-08-1748 edits
Open-source Text-To-Image similar to DALL-E
Stable Diffusion is a machine learning, text-to-image model developed by StabilityAI, in collaboration with EleutherAI and LAION, to generate digital images from natural language descriptions. The model can be used for other tasks too, like generating image-to-image translations guided by a text prompt.
Features
- Latent Diffusion model trained on 512x512 images from the LAION-5B database
- Uses the CLIP ViT-L/14 text encoder, similar to Google's Imagen
- Lightweight, so it runs even on a single GPU with 10GB+ VRAM
- Collaboration and sponsorship by Stability AI and LAION researchers
Training methods
- The guide to fine-tuning Stable Diffusion with your own images | Tryolabs
- Stable Diffusion text-to-image fine-tuning
- lambdalabs/pokemon-blip-captions · Datasets at Hugging Face
Training the model directly
Textual Inversion
A method that creates an embedding file. See Textual Inversion for details.
Dreambooth
A method that trains the model on 200 similar images to modify it. See Dreambooth for details.
LoRA
See LoRA for details.
Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models
- How to generate "a ○○ like ~" from just one image in "Stable Diffusion" in only a few tens of seconds
- Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models
Tools
- Draw Anything - A simple Stable Diffusion Playground
- https://www.drawanything.app
- A tool where you draw or edit an image, add more description with a prompt, and let the Stable Diffusion AI finish it
- You can feed a created image back in as input to keep developing it
- (Google or email login required to load previously worked images)
- Images found through the internal search can also be used
- Stable Diffusion Web UI
- https://github.com/AUTOMATIC1111/stable-diffusion-webui/
-
https://github.com/sd-webui/stable-diffusion-webui
- InvokeAI - Stable Diffusion toolkit & WebGUI
- New name of the former Stable Diffusion Dream Script
- Completely new WebGUI
- Supports inpainting & outpainting
- Improved in-app workflows such as GFPGAN, Codeformer, and RealESRGAN upscaling
- Windows/Mac/Linux
- Runs on NVIDIA graphics cards with 4+GB VRAM or on Apple M1 & M2
- At least 12GB of main RAM
- Gauss
- Open-source native Stable Diffusion app for macOS
- Img2Prompt - Inferring prompts from images generated with Stable Diffusion
- https://www.img2prompt.io/
- A tool that infers a text prompt from an image made by generative AI
- You can try producing related prompts even for images not made by AI
- Trained on images generated with Stable Diffusion 1.4/5
- Comments The training dataset itself isn't very large, so it isn't accurate, but it seems worth trying just for fun.
- Web Stable Diffusion
- Running SD in a web browser
- Runs Stable Diffusion with WebGPU in the browser alone, without a server
Projects
- Stable Diffusion-based level editor for 2D games
- https://generalrobots.substack.com/p/dimension-hopper-part-1
- The journey of a project to make a 2D platformer game where players design their own levels and AI generates the rendered images.
- Tried generating from images of drawn blocks using ControlNet Depth mode on a model trained on pixel images, but the results were poor.
- Switched to Scribble mode and kept refining the prompt to get somewhat better results.
- Since the tiles did not need to repeat, switched from the pixel model to a children's illustration model and got non-repeating, painting-like images.
- The results looked good, but because of the limits of Scribble mode, too many pictures came out where you could not tell where the ground was.
- Went back to Depth mode, but after additionally drawing white platforms on top of the blocks, got images where the ground the player can stand on was always clearly distinguishable.
- However, even with prompt adjustments, only monotonous backgrounds came out.
- To solve this, separated all the elements, such as platforms, blocks, block pillars, and the background, and adjusted their depth, which produced much higher-quality images.
- Finally, changing the blocks from squares to bumpy shapes made the ground non-square, giving final images that look like a real world.
- Since there is a depth map, split the world based on the depth map and placed objects and the player at suitable positions.
- Combined all of this and released it as a real game.
- Comic-Factory - A tool that generates comics using diffusion models
- AI Comic Factory - a Hugging Face Space by jbilcke-hf
- Generates comics in 4-panel units using comic style presets and a short story prompt
- Presets: Japanese (published manga), American (graphic novel), Franco-Belgian (French/Belgian style), Armorican (ancient art style)
Prompts
- MagicPrompt-Stable-Diffusion
- Prompt generator for SD
- A GPT-2 model that creates prompts for image generation AI
- Enter a prompt for the picture you want, and it writes an SD prompt that generates a better-quality image
- MagicPrompt also has separate versions for Dall-E 2 and Midjourney
- PromptBase - Prompt marketplace
- You can buy the 'prompt' itself that generated a specific image or text with AI
- You look at the generated image and buy the prompt text
- DALL·E / Midjourney / StableDiffusion / GPT-3
- You can also generate with Stable Diffusion yourself and sell that prompt right away
- Lexica - Search over 5 million Stable Diffusion-generated images
- Search images and prompts generated with Stable Diffusion
- In the prompts found by a keyword search, click part of the prompt text to drill down into a more detailed search
- Copy prompt / copy image URL
- Open Prompts - Dataset of 10 million Stable Diffusion prompts (github.com/krea-ai)
- Releases the entire dataset of prompts generated on the Stability AI Discord server during the Stable Diffusion v1.3 beta test
- CSV files include the prompt text and links to the images
- Also searchable through a REST-style Prompts API
- It was used to build Krea.ai, a service for searching AI-generated images, and the service also provides the data in category form
- Home - DiffusionDB
- Large-scale text-to-image prompt gallery dataset based on Stable Diffusion
- (arxiv) DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
- https://github.com/poloclub/diffusiondb
- Civitai
- Stable Diffusion models, embeddings, hypernetworks and more
See also
- Diffusion model (Korean: 디퓨전 모델; Korean: 확산 모델)
- Stability AI
- StableLM
- Deep learning
- Machine learning
- Text-To-Image
- Imagen - Google's text-to-image diffusion model
- DALL-E
- Creating Moe Illustrations Using AI Tutorial - Tutorial on creating moe illustrations using AI
- Midjourney
- HuggingFace
- Scene Diffuser - Diffusion-based 3D scene generation
- DeepFloyd IF - Open-source Text-to-Image model
- Opendream - Layer-based UI for Stable Diffusion
- Fooocus - Easier and more convenient image generation software that learned from SD and Midjourney
- ComfyUI - A powerful and modular GUI for Stable Diffusion models. Enables node graph-based visual programming. - You can test models downloaded from civitai.com.
- Mercury - Diffusion-based ultra-fast language model
- Retro Diffusion - Generate Pixel Art with AI
- Mercury 2 - Diffusion-based ultra-fast reasoning LLM
Favorite site
- Stable Diffusion XL 1.0 (SDXL), released as open source, supports generating 1024x1024 resolution images
- SDXL comes as two models, Base and Refine; the Refine model causes no slowdown, so it is better to use it
- Hackable with the Diffusers Python library
- Features such as Prompt Weighting and Dreambooth LoRA are possible
- Training LoRA on bad images with a "wrong" negative prompt improved quality
huggingface
References
-
The_Illustrated_Stable_Diffusion_-Jay_Alammar-_Visualizing_machine_learning_one_concept_at_a_time.pdf ↩