Stanford CRFM
lastmod 2025-06-01created 2025-06-011 edits
AI-generated CUDA kernels surpass the performance of PyTorch-optimized code
Features
- AI-generated CUDA-C kernels match or outperform PyTorch's expert-optimized kernels
- A single LLM (large language model) iterates between generating natural-language optimization ideas and branching into various code variants, outperforming existing methods in optimization diversity and parallel search
- In representative benchmarks, it overwhelmingly beats PyTorch on Matmul (101%), Conv2D (179.9%), Softmax (111.8%), LayerNorm (484.4%), Conv2D+ReLU+MaxPool (290.1%), and more
- Applies natural-language reasoning plus a branching structure to overcome the limits of conventional "sequential kernel improvement", generating high-performance kernels quickly
- Also making progress on modern ML workloads such as FP16 and Flash Attention, suggesting that a paradigm where AI searches for and improves faster kernels on its own may become mainstream in the future