GPU MODE Lecture Notes – Christian Mills

Modified

September 24, 2024

Resource Links:

Discord Channel: GPU MODE
YouTube Channel: GPU MODE
GitHub Repository: GPU MODE Lectures
Textbook: Programming Massively Parallel Processors

GPU MODE Lecture 1: How to profile CUDA kernels in PyTorch

notes

cuda

pytorch

Lecture #1 provides a practical introduction to integrating and profiling custom CUDA kernels within PyTorch programs, using tools like load_inline, Triton, and NVIDIA Nsight Compute.

GPU MODE Lecture 2: Ch.1-3 PMPP Book

notes

cuda

pytorch

Lecture #2 provides an introduction to parallel programming with CUDA C, covering key concepts like heterogeneous computing, data parallelism, thread organization, and memory management, and showcasing examples such as vector addition, image blurring, and matrix multiplication.

GPU MODE Lecture 3: Getting Started With CUDA for Python Programmers

notes

cuda

pytorch

Lecture #3 provides a beginner-friendly introduction to CUDA programming with PyTorch, demonstrating how to write and execute CUDA kernels within a Python environment for tasks like image processing and matrix multiplication.

GPU MODE Lecture 4: Compute and Memory Basics

notes

cuda

Lecture #4 provides an overview of CUDA programming fundamentals, focusing on compute architecture, memory management, and optimization strategies like kernel fusion and tiling to achieve high-performance GPU computation.

GPU MODE Lecture 5: Going Further with CUDA for Python Programmers

notes

cuda

Lecture #5 explores how to optimize matrix multiplication in CUDA for Python programmers using shared memory and tiling, comparing implementations in pure Python, CUDA C, and the Numba library.

GPU MODE Lecture 6: Optimizing Optimizers in PyTorch

notes

cuda

Lecture #6 explores PyTorch’s optimization strategies for speeding up optimizers, focusing on techniques like kernel fusion and multi-tensor apply to reduce kernel launch overhead and improve runtime performance.

GPU MODE Lecture 7: Advanced Quantization

notes

cuda

Lecture #7 discusses GPU quantization techniques in PyTorch, focusing on performance optimizations using Triton and CUDA kernels for dynamic and weight-only quantization, including challenges and future directions.

GPU MODE Lecture 8: CUDA Performance Checklist

notes

cuda

Lecture #8 provides a comprehensive guide to CUDA performance optimization techniques, covering key concepts like memory coalescing, occupancy, control divergence, tiling, privatization, thread coarsening, and algorithm rewriting with better math, illustrated with practical examples and profiling using NCU to improve kernel performance.

Notes on Trends in Deep Learning Hardware: Bill Dally (NVIDIA)

notes

cuda

In this lecture, Bill Dally discusses the historical progress of deep learning, driven by hardware advancements, especially GPUs, and explores future directions focusing on improving performance and efficiency through techniques like optimized number representation, sparsity, and specialized hardware accelerators.

GPU MODE Lecture 9: Reductions

notes

cuda

Lecture #9 covers parallel reduction algorithms for GPUs, focusing on optimizing their implementation in CUDA by addressing control divergence, memory divergence, minimizing global memory accesses, and thread coarsening, ultimately demonstrating how these techniques are employed in machine learning frameworks like PyTorch and Triton.

GPU MODE Lecture 10: Build a Prod Ready CUDA library

notes

cuda

Lecture #10 explores the challenges of real-time CUDA development, focusing on building a C++ library for efficient GPU communication and task scheduling to abstract away CUDA complexities for non-expert users while maintaining performance.

GPU MODE Lecture 11: Sparsity

notes

cuda

Lecture #11 discusses GPU sparsity, specifically semi-structured and block sparsity techniques, for accelerating neural network inference and training by leveraging optimized kernels and sparse data representations.

GPU MODE Lecture 12: Flash Attention

notes

cuda

Lecture #12 provides an introduction to Flash Attention, a highly optimized CUDA kernel for accelerating attention computations in transformer models, including a conceptual overview, tiling strategy, softmax stabilization, and limitations.

GPU MODE Lecture 13: Ring Attention

notes

cuda

Lecture #13 explores ring attention, a distributed computing technique for training long-context transformers, discussing its motivation and underlying mechanisms.

GPU MODE Lecture 14: Practitioners Guide to Triton

notes

cuda

Lecture #14 provides a practical introduction to writing and optimizing GPU kernels using Triton, featuring comparisons with CUDA and hands-on examples like tensor copying, image processing, and matrix multiplication.

GPU MODE Lecture 15: CUTLASS

notes

cuda

Lecture #15 provides an in-depth conceptual explanation of NVIDIA’s Cutlass library’s tensor layout algebra system, focusing on how it handles shapes, strides, and tiling operations for efficient GPU tensor computations.