GPU MODE Lecture Notes

My notes from the GPU MODE (formerly CUDA MODE) reading group lectures run by Andreas Kopf and Mark Saroufim.
Modified

October 14, 2025

NoteResource Links:

GPU MODE Lecture 1: How to profile CUDA kernels in PyTorch

notes
cuda
pytorch

Lecture #1 provides a practical introduction to integrating and profiling custom CUDA kernels within PyTorch programs, using tools like load_inline, Triton, and NVIDIA Nsight Compute.

Apr 26, 2024
51 min

GPU MODE Lecture 2: Ch.1-3 PMPP Book

notes
cuda
pytorch

Lecture #2 provides an introduction to parallel programming with CUDA C, covering key concepts like heterogeneous computing, data parallelism, thread organization, and memory management, and showcasing examples such as vector addition, image blurring, and matrix multiplication.

Jun 6, 2024
27 min

GPU MODE Lecture 3: Getting Started With CUDA for Python Programmers

notes
cuda
pytorch

Lecture #3 provides a beginner-friendly introduction to CUDA programming with PyTorch, demonstrating how to write and execute CUDA kernels within a Python environment for tasks like image processing and matrix multiplication.

Aug 31, 2024
49 min

GPU MODE Lecture 4: Compute and Memory Basics

notes
cuda

Lecture #4 provides an overview of CUDA programming fundamentals, focusing on compute architecture, memory management, and optimization strategies like kernel fusion and tiling to achieve high-performance GPU computation.

Aug 31, 2024
52 min

GPU MODE Lecture 5: Going Further with CUDA for Python Programmers

notes
cuda

Lecture #5 explores how to optimize matrix multiplication in CUDA for Python programmers using shared memory and tiling, comparing implementations in pure Python, CUDA C, and the Numba library.

Sep 1, 2024
57 min

GPU MODE Lecture 6: Optimizing Optimizers in PyTorch

notes
cuda

Lecture #6 explores PyTorch’s optimization strategies for speeding up optimizers, focusing on techniques like kernel fusion and multi-tensor apply to reduce kernel launch overhead and improve runtime performance.

Sep 2, 2024
12 min

GPU MODE Lecture 7: Advanced Quantization

notes
cuda

Lecture #7 discusses GPU quantization techniques in PyTorch, focusing on performance optimizations using Triton and CUDA kernels for dynamic and weight-only quantization, including challenges and future directions.

Sep 10, 2024
14 min

GPU MODE Lecture 8: CUDA Performance Checklist

notes
cuda

Lecture #8 provides a comprehensive guide to CUDA performance optimization techniques, covering key concepts like memory coalescing, occupancy, control divergence, tiling, privatization, thread coarsening, and algorithm rewriting with better math, illustrated with practical examples and profiling using NCU to improve kernel performance.

Sep 11, 2024
68 min

Notes on Trends in Deep Learning Hardware: Bill Dally (NVIDIA)

notes
cuda

In this lecture, Bill Dally discusses the historical progress of deep learning, driven by hardware advancements, especially GPUs, and explores future directions focusing on improving performance and efficiency through techniques like optimized number representation, sparsity, and specialized hardware accelerators.

Sep 11, 2024
17 min

GPU MODE Lecture 9: Reductions

notes
cuda

Lecture #9 covers parallel reduction algorithms for GPUs, focusing on optimizing their implementation in CUDA by addressing control divergence, memory divergence, minimizing global memory accesses, and thread coarsening, ultimately demonstrating how these techniques are employed in machine learning frameworks like PyTorch and Triton.

Sep 14, 2024
53 min

GPU MODE Lecture 10: Build a Prod Ready CUDA library

notes
cuda

Lecture #10 explores the challenges of real-time CUDA development, focusing on building a C++ library for efficient GPU communication and task scheduling to abstract away CUDA complexities for non-expert users while maintaining performance.

Sep 15, 2024
17 min

GPU MODE Lecture 11: Sparsity

notes
cuda

Lecture #11 discusses GPU sparsity, specifically semi-structured and block sparsity techniques, for accelerating neural network inference and training by leveraging optimized kernels and sparse data representations.

Sep 15, 2024
11 min

GPU MODE Lecture 12: Flash Attention

notes
cuda

Lecture #12 provides an introduction to Flash Attention, a highly optimized CUDA kernel for accelerating attention computations in transformer models, including a conceptual overview, tiling strategy, softmax stabilization, and limitations.

Sep 15, 2024
27 min

GPU MODE Lecture 13: Ring Attention

notes
cuda

Lecture #13 explores ring attention, a distributed computing technique for training long-context transformers, discussing its motivation and underlying mechanisms.

Nov 11, 2024
16 min

GPU MODE Lecture 14: Practitioners Guide to Triton

notes
cuda

Lecture #14 provides a practical introduction to writing and optimizing GPU kernels using Triton, featuring comparisons with CUDA and hands-on examples like tensor copying, image processing, and matrix multiplication.

Nov 15, 2024
72 min

GPU MODE Lecture 15: CUTLASS

notes
cuda

Lecture #15 provides an in-depth conceptual explanation of NVIDIA’s Cutlass library’s tensor layout algebra system, focusing on how it handles shapes, strides, and tiling operations for efficient GPU tensor computations.

Nov 17, 2024
19 min
No matching items