Skip to content
Search

Bolt3D

lastmod 2025-03-22created 2025-03-223 edits

Ultra-fast 3D scene generation model

About

  • A latent diffusion model for ultra-fast 3D scene generation
  • Can generate a high-resolution 3D scene from one or more input images in under 7 seconds
  • Trained on a purpose-built large-scale multi-view-consistent dataset; up to 300x faster than existing 3D generative models
  • Existing models require an optimization process, but Bolt3D can generate scenes instantly in a feed-forward manner

Bolt3D_-_howitworks_website.jpg

Limitations and problems of existing models

  • Existing 2D generative models can generate high-quality images, but 3D scene generation is difficult
  • Existing 3D models have the following problems:
    • Difficulty handling complex 3D data structures
    • Lack of high-quality real 3D scene data
    • High computational cost and slow processing speed

Key techniques and structure of Bolt3D

3D representation

  • Uses a 3D Gaussian representation:
    • A 3D Gaussian consists of color, position, opacity, and a covariance matrix
    • 3D Gaussian rendering is performed via pixel-aligned images called Splatter Images
    • Can also fill in unseen regions

Bolt3D generation process

  • Estimates the 3D scene from input images via a latent diffusion model
  • Encodes geometric information into the latent space via the Geometry VAE
  • The Gaussian Head predicts and corrects the detailed attributes of the 3D Gaussians (opacity, color, etc.)
  • Performs instant rendering of high-resolution 3D scenes

Model structure

  • The latent diffusion model adopts a structure evolved from 2D image generative models
  • The Geometry VAE encodes 3D pointmaps and camera poses
  • The Gaussian Head refines the detailed attributes of the generated 3D scene

Dataset and training

  • Built a large-scale multi-view dataset:
    • Includes CO3D, MVImg, RealEstate10K, DL3DV-7K
    • Consists of about 300,000 multi-view scenes in total
    • Uses the MASt3R technique to obtain accurate geometric data
  • Training process:
    • Geometry VAE: trained at 256×256 → 512×512 resolution
    • Gaussian Head: refines Splatter Image generation
    • Latent Diffusion Model: fine-tuned from the CAT3D model

Experimental results and performance comparison

Comparison with existing models

  • Bolt3D outperforms the existing Flash3D and DepthSplat models
  • Compared with Flash3D, Bolt3D scored about 3.6 points higher on PSNR, and also improved on SSIM and LPIPS
  • Compared with the DepthSplat model, Bolt3D also came out ahead on all performance metrics
  • The performance gain was largest especially when there was only one input image

Performance comparison with optimization-based models

  • Compared with existing optimization-based models such as CAT3D, Bolt3D achieved similar or better performance while being 300x faster
  • CAT3D takes about 5 minutes to generate a scene, while Bolt3D can perform the same task in 6.25 seconds
  • On performance metrics, CAT3D scored a somewhat higher PSNR than Bolt3D, but Bolt3D was overwhelmingly better in processing speed

Model structure and architecture improvements

  • Geometry VAE improvements
    • Uses a VAE dedicated to geometric information → higher accuracy than a general image VAE
    • Applies nonlinear scaling and depth mapping → improved model performance
  • Gaussian Head improvements
    • Integrates and corrects multi-view information
    • Applies Cross-Attention → can also fill in unseen regions
  • Conclusions and implications
    • Bolt3D enables fast 3D scene generation through learning geometric information and a feed-forward approach
    • Both performance and speed are improved over existing models
    • Enables instant high-quality 3D scene generation in various application areas:
      • Game development
      • Virtual reality (VR) and augmented reality (AR)
      • Architecture and design visualization
    • With processing speed improved 300x per second, it has high potential for commercialization and scaling

Summary of key achievements

  • Can generate 3D scenes in under 7 seconds
  • 300x faster than existing models
  • High-resolution detail and consistency
  • High performance with single and multiple views
  • Can naturally fill in even complex and incomplete scenes

Documentation

[2503.14445] Bolt3D - Generating 3D Scenes in Seconds
https://arxiv.org/abs/2503.14445

See also

Favorite site

author Zer0 sitemap