Skip to content
Search

GPU

lastmod 2026-03-12created 2023-08-247 edits

A graphics processing unit (GPU) is a processing unit in a computer system that quickly processes graphics operations and outputs the results to a monitor. It is also called a VPU (visual processing unit).

GPUs are used in embedded systems, mobile phones, personal computers, workstations, video game consoles, artificial intelligence, driverless cars, cloud computing, and more. Modern GPUs are very effective at computer graphics and image processing, and their highly parallel structure makes them more efficient than general-purpose CPUs for algorithms that process large blocks of image data in parallel. In a personal computer, the GPU is mounted on a graphics card, and depending on the motherboard or CPU, it may also be included on the die.

The term 'GPU' was popularized in 1999 when NVIDIA marketed the GeForce 256 as 'the world's first GPU'. The GeForce 256 had a single-chip processor with integrated TCL (Transform, clipping, and lighting) and a rendering engine capable of processing more than 10 million polygons per second. Rival ATI Technologies used the term 'VPU' (visual processing unit) when it released the Radeon 9700 in 2002.

A GPU can greatly reduce the CPU's load for graphics-related computation. macOS was the first to start using the GPU at the operating system level, starting with 10.6.

Cloud GPU guide - which GPU should you use for AI, and where?

If you build AI tools using StableDiffusion, Whisper, open-source LLMs, etc., they need to keep running somewhere.

Which GPU should you use?

If you use cloud GPUs

  • Falcon-40B, Falcon-40B-Uncensored, or Falcon-40B-Instruct
    • For the best performance regardless of cost: 2x H100
    • For a balance of cost and performance: 2x RTX 6000 Ada (not A6000 or RTX6000)
    • For cheap: 2x A6000
  • MPT-30B
    • Best performance or value: 1x H100
    • Cheap: 1x A100 80GB
  • Stable Diffusion
    • Best performance: 1x H100
    • Value: 1x 4090
    • Cheap: 1x 3090
  • Whisper
    • Same as Stable Diffusion
    • Whisper-Large can run on less VRAM, but most clouds don't have such cards
    • 4090/3090 also work well, and it can run on CPU too
  • If you fine-tune large LLMs
    • H100 cluster or A100 cluster
  • If you want to train large LLMs
    • Large H100 cluster
    • If you use local GPUs:
  • Mostly the same as above, but LLM training and fine-tuning are not possible
  • Most LLMs have versions that can run on small VRAM (Falcon on 40GB)

Should you run models locally or on cloud GPUs?

  • Both are reasonable choices
  • To run models in the cloud, Runpod templates are the easiest choice
  • The easiest option is to use hosted instances: DreamStudio, RunDiffusion, Playground AI for stable diffusion, etc.

What's the difference between RTX 6000, A6000, and 6000 Ada?

The three are completely different

  • RTX 6000 (Quadro RTX 6000, 24 GB VRAM, released 2018/08/13)
  • RTX A6000 (48 GB VRAM, released 2020/10/05)
  • RTX 6000 Ada (48 GB VRAM, released 2022/12/03)

DGX GH200, GH200, H100?

  • One DGX GH200 contains 256 GH200s
  • One GH200 contains one H100 and one Grace CPU

Is the H100 a big upgrade over the A100?

  • Of course. The speedup is very large. H100 can scale to more GPUs than A100
  • So multiple H100s are the best for LLM training

What about AMD, Intel, Cerebras?

For now, Nvidia is the easiest

Which GPU cloud should you use?

  • If you need many A100s/H100s: contact Oracle, FluidStack, Lambda Labs, etc.
  • If you need a few A100s: FluidStack or Runpod
  • If you need one H100: FluidStack or Lambda Labs
  • Cheap 3090s, 4090s, A6000s: Tensordock.
  • If you only need Stable Diffusion inference: Salad.
  • If you need many kinds of GPUs: Runpod or FluidStack.
  • If you use templates or do it as a hobby: Runpod.
  • Big clouds are expensive and complicated

Easiest GPU Cloud to get started with

  • Use templates on RunPod
  • Note that RunPod pods are Docker containers on a host machine, not full-featured VMs

How much VRAM and system RAM, and how many vCPUs, do you need?

  • VRAM (Video RAM / GPU RAM)
    • Falcon-40B: 85-100GB
    • MPT-30B: 80GB
    • Stable Diffusion: 16GB+ or more preferred
    • Whisper: 12GB+. (About this much with the OpenAI version; community versions can run on CPU)
  • System ram
    • 1-2x the VRAM
  • vCPUs
    • 8-16 vCPUs are enough unless it's a large GPU workload
  • Disk space
  • Depends on the use case. If unsure, start with 100GB and see if it fits your use case

Basic facts about GPUs

  • Basic facts about GPUs | GeekNews
  • On GPUs, computation is far faster than memory access, so the memory hierarchy becomes the performance bottleneck
  • Depending on Arithmetic Intensity (AI), operations are classified as memory-bound or compute-bound; the threshold for the A100 GPU is about 13 FLOPs/Byte
  • Key performance optimization strategies are operation fusion (Fusion) and tiling (Tiling); Fusion reduces unnecessary memory round trips, and Tiling maximizes data reuse
  • Understanding the structural characteristics of GPU hardware, such as synchronization, Coalesced Load, and resolving bank conflicts, is important for writing high-performance kernels
  • Additional considerations such as occupancy (Occupancy), minimizing thread divergence, and quantization (Quantization) have a significant impact on real performance

See also

Favorite site

Online Tools

  • [Recommended] LeetGPU - Only platform to write and run CUDA code. Without a GPU. For Free. (a site for running CUDA code online for free)
author Zer0 sitemap