Riffusion

lastmod 2023-02-01created 2023-02-012 edits

Stable Diffusion fine-tuned to generate music

Features

• Uses SD 1.5 as is, fine-tuning the model to generate Spectrogram images using img2txt
• The generated images can be converted into playable audio clips
• Sound can be modified using img2img
• Interpolation so that short audio clips can loop infinitely
• Built as a web app for ease of use, with the code also released (Next.js + React + Three.js + Tailwind)
• Spectrogram : Waveform + Spectrum
  • A visual representation of the spectrum of a sound using frequency, amplitude (intensity), and time
  • Can be computed from audio using STFT (Short-Time Fourier Transform)

See also

• Deep learning
• Magenta
• DeepJazz
• BachBot
• FlowMachines
• WaveNet
• GRUV
• MuseNet
• MusicLM - AI that generates music from text
• Riffusion - Stable Diffusion fine-tuned to generate music

Favorite site

• Riffusion