From the free course Introduction to CUDA and Modern C++ Programming on Lykke

The CUDA Programming Model & Architecture

Key concepts: Kernels · Thread Hierarchy · Memory Hierarchy · SIMT Architecture · NVCC Compilation

An introduction to the fundamental concepts of GPU computing, focusing on the thread hierarchy, memory models, and the SIMT architecture.

The CUDA Programming Model & Architecture

The emergence of General-Purpose computing on Graphics Processing Units (GPGPU) represents one of the most significant shifts in high-performance computing (HPC) over the last two decades. At the heart of this revolution is CUDA (Compute Unified Device Architecture), NVIDIA’s parallel computing platform and programming model.

CUDA is not merely a language extension; it is a comprehensive hardware-software co-design. It exposes the GPU’s massive parallelism through a familiar C++ syntax while abstracting the underlying complexities of many-core hardware. To master CUDA, one must move beyond thinking in terms of sequential loops and embrace a hierarchy of threads, memories, and execution units that operate under the SIMT (Single Instruction, Multiple Threads) paradigm.

Kernels: The Entry Point to Parallelism

In the CUDA programming model, the Kernel is the fundamental unit of execution. Unlike a standard C++ function that executes once when called, a CUDA kernel is a function that, when invoked, is executed $N$ times in parallel by $N$ different CUDA threads.

What it is

A kernel is defined using the __global__ declaration specifier. The execution configuration for a kernel is defined at call time using the execution configuration syntax <<<...>>>, which specifies the number of threads and their organization.

Definition: The CUDA Kernel A kernel is a C++ function that executes on the device (GPU). It is characterized by its ability to perform the same operation on different subsets of data simultaneously, driven by unique coordinates provided by the hardware.

Why it matters

Kernels allow developers to express data-level parallelism explicitly. By offloading data-intensive portions of an application to the GPU, the CPU (host) is freed to handle sequential logic and I/O, while the GPU (device) handles the "heavy lifting" of throughput-oriented tasks.

How it works: The Execution Configuration

The <<<G, B>>> syntax defines the Grid (G) and Block (B) dimensions. Each thread executing the kernel is assigned a unique thread ID that it uses to compute memory addresses and make control-flow decisions.

// Block 1: Low-level CUDA C++ Implementation
// Example: Tiled Matrix Multiplication using Shared Memory
// This demonstrates kernel definition, shared memory usage, and synchronization.

#include <cuda_runtime.h>
#include <iostream>

#define TILE_WIDTH 16

__global__ void MatrixMulKernel(float* d_A, float* d_B, float* d_C, int Width) {
    // Allocate shared memory for tiles
    __shared__ float ds_A[TILE_WIDTH][TILE_WIDTH];
    __shared__ float ds_B[TILE_WIDTH][TILE_WIDTH];

    int bx = blockIdx.x;  int by = blockIdx.y;
    int tx = threadIdx.x; int ty = threadIdx.y;

    // Identify the row and column of the d_C element to work on
    int Row = by * TILE_WIDTH + ty;
    int Col = bx * TILE_WIDTH + tx;

    float Pvalue = 0;

    // Loop over the d_A and d_B tiles required to compute the d_C element
    for (int m = 0; m < (Width / TILE_WIDTH); ++m) {
        // Collaborative loading of tiles into shared memory
        ds_A[ty][tx] = d_A[Row * Width + (m * TILE_WIDTH + tx)];
        ds_B[ty][tx] = d_B[(m * TILE_WIDTH + ty) * Width + Col];

        // Synchronize to ensure all threads have finished loading the tile
        __syncthreads();

        for (int k = 0; k < TILE_WIDTH; ++k) {
            Pvalue += ds_A[ty][k] * ds_B[k][tx];
        }

        // Synchronize to ensure computation is done before loading next tile
        __syncthreads();
    }
    d_C[Row * Width + Col] = Pvalue;
}

Thread Hierarchy: Organizing Massive Parallelism

CUDA threads are not flat; they are organized into a three-level hierarchy to match the physical architecture of the GPU. This hierarchy consists of Threads, Blocks, and Grids.

The Hierarchy Levels

Level Description Max Dimensions Scope
Thread The smallest unit of execution. N/A Local registers and private memory.
Block A group of threads that can cooperate via shared memory and barriers. (1024, 1024, 64) Shared memory within the block.
Grid A collection of blocks that execute the same kernel. ($2^{31}-1$, 65535, 65535) Global memory across the whole GPU.

Thread Indexing Math

To map a thread to a specific data element, we use built-in variables: threadIdx, blockIdx, and blockDim. For a 1D grid of 1D blocks, the unique global index $i$ is calculated as:

i = blockIdx.x \times blockDim.x + threadIdx.x

For a 2D grid, the derivation becomes more complex, mapping a 2D coordinate $(x, y)$ to a linear memory address.

// Block 2: Mathematical Derivation of Indexing
// Mapping a 2D thread hierarchy to a 1D linear array index

Given:
  - threadIdx.x, threadIdx.y (Thread coordinates within a block)
  - blockIdx.x,  blockIdx.y  (Block coordinates within a grid)
  - blockDim.x,  blockDim.y  (Dimensions of a block)
  - gridDim.x                (Number of blocks in the x-direction)

The global column index (j):
  j = blockIdx.x * blockDim.x + threadIdx.x

The global row index (i):
  i = blockIdx.y * blockDim.y + threadIdx.y

The linear index (Idx) for a matrix of width 'W':
  Idx = i * W + j

Proof of Uniqueness:
  Since 0 <= threadIdx.x < blockDim.x 
  And 0 <= blockIdx.x < gridDim.x
  The range of 'j' is [0, gridDim.x * blockDim.x - 1].
  Every combination of (blockIdx, threadIdx) maps to a unique integer.

AI_DEMOI_DEMO## Memory Hierarchy: Managing Latency and Bandwidth

A common mistake among novice CUDA developers is treating GPU memory as a monolithic block. In reality, CUDA defines several distinct memory spaces, each with different capacities, latencies, and caching behaviors.

Memory Types and Characteristics

Memory Location Cached Access Scope Lifetime
Register On-chip No R/W 1 Thread Thread
Local Off-chip Yes R/W 1 Thread Thread
Shared On-chip N/A R/W All threads in block Block
Global Off-chip Yes R/W All threads + Host Application
Constant Off-chip Yes R All threads + Host Application

Shared Memory and Bank Conflicts

Shared Memory is a high-bandwidth, low-latency memory space located on the Streaming Multiprocessor (SM). It is effectively a user-managed L1 cache. However, it is divided into 32 equally sized modules called banks. If multiple threads in a warp attempt to access different addresses that map to the same bank, a bank conflict occurs, and the accesses are serialized.

Theorem: The Conflict-Free Condition A shared memory access is conflict-free if every thread in a warp accesses an address in a different bank, or if all threads access the same address (broadcast).

SIMT Architecture: Single Instruction, Multiple Threads

The GPU hardware executes threads using the SIMT (Single Instruction, Multiple Threads) architecture. While similar to SIMD (Single Instruction, Multiple Data) found in CPUs (like AVX), SIMT is more flexible.

Warps: The Unit of Execution

In SIMT, the hardware groups 32 threads into a unit called a Warp. All threads in a warp start together at the same program address, but each has its own instruction address counter and register state. This allows them to branch independently.

Branch Divergence

When threads within a warp take different paths in a conditional statement (if-else), the warp serially executes each branch path, disabling threads that are not on that path. This is known as Branch Divergence and can significantly degrade performance.

Coalescing Global Memory

When threads in a warp access global memory, the hardware attempts to combine these into a single memory transaction. This is called Coalescing. If thread $i$ accesses address $Base + i$, the hardware can fetch the data for the entire warp in one go. If the access pattern is scattered, the hardware must issue multiple transactions, wasting bandwidth.

NVCC Compilation: The Path from Source to Binary

Compiling CUDA code is more complex than standard C++ because the source file (.cu) contains both CPU code (Host) and GPU code (Device). NVIDIA provides the nvcc compiler driver to handle this.

The Compilation Pipeline

  1. Separation: nvcc separates the host code from the device code.
  2. Device Compilation: The device code is compiled into an assembly-like intermediate representation called PTX (Parallel Thread Execution).
  3. Host Compilation: The host code is modified to replace kernel calls with the CUDA Runtime function calls necessary to launch the kernel.
  4. Assembly: The PTX is further compiled into SASS (Streaming Assembler), which is the machine code specific to a particular GPU architecture (e.g., Ampere, Hopper).

Fatbinaries

nvcc can produce fatbinaries, which contain both PTX and SASS for multiple GPU architectures. This ensures that the application can run on older GPUs (via SASS) and future GPUs (by JIT-compiling the PTX at runtime).

# Block 3: Real-world Usage - CLI and Build Pipeline
# Compiling for multiple architectures and profiling

# Compile for Compute Capability 8.0 (Ampere) and 9.0 (Hopper)
# -gencode generates SASS for specific architectures
# -ptx generates the intermediate PTX file for inspection
nvcc -O3 -arch=sm_80 \
    -gencode arch=compute_80,code=sm_80 \
    -gencode arch=compute_90,code=sm_90 \
    -Xcompiler -Wall \
    main.cu -o matrix_mul_app

# Profile the application to check for occupancy and memory throughput
nsys profile --stats=true ./matrix_mul_app

# Use ncu (NVIDIA Compute Profiler) to look for bank conflicts in the kernel
ncu --metrics l1tex__data_bank_conflicts_pipe_lsu_mem_shared_op_ld.sum ./matrix_mul_app

Modern CUDA: Variadic Templates and Type Deduction

As the C++ standard evolved (C++11, 14, 17, 20), CUDA adopted many of its features. This allows for much more expressive and generic parallel code.

Variadic Kernels

Using variadic templates (as proposed in N2242), developers can create generic kernel launchers that accept an arbitrary number of arguments, significantly reducing boilerplate code.

Auto and Type Deduction

The redefinition of auto (N1984) is particularly useful in CUDA for handling complex iterator types or results of template-heavy math libraries like thrust or cutlass.

// Block 4: Modern CUDA C++ - Variadic Template Kernel Launcher
// Demonstrating how C++11/14 features simplify CUDA development

template<typename... Args>
__global__ void generic_kernel(Args... args) {
    // Pack expansion to process arguments
    auto process = [](auto val) { /* ... do something ... */ };
    (process(args), ...); // Fold expression (C++17)
}

template<typename... Args>
void launch_helper(int blocks, int threads, Args... args) {
    // Using auto for the grid/block dimensions
    auto grid = dim3(blocks);
    auto block = dim3(threads);
    
    // Launching the variadic kernel
    generic_kernel<<<grid, block>>>(args...);
    
    // Check for launch errors
    cudaError_t err = cudaGetLastError();
    if (err != cudaSuccess) {
        std::cerr << "CUDA Error: " << cudaGetErrorString(err) << std::endl;
    }
}

Common Pitfalls and Best Practices

  1. Ignoring Error Codes: CUDA kernel launches are asynchronous. Always check cudaGetLastError() and the return values of API calls like cudaMalloc.
  2. Over-using Global Memory: Global memory is slow. If you access the same data multiple times within a block, load it into Shared Memory first.
  3. Low Occupancy: If your kernel uses too many registers or too much shared memory per thread, the hardware cannot schedule enough warps to hide memory latency. This is known as low Occupancy.
  4. Host-Device Synchronization: Frequent transfers between CPU and GPU memory (cudaMemcpy) are the most common bottleneck. Keep data on the GPU as long as possible.
  5. Thread Divergence in Loops: Be careful with loops where different threads might iterate a different number of times; this causes the same performance penalty as a divergent if statement.

AI_FLASHCARDSI_FLASHCARDS Kernel: A GPU-executed function marked with __global__.

  • Warp: A group of 32 threads executing in SIMT fashion.
  • Shared Memory: On-chip, user-managed cache shared by threads in a block.
  • Coalescing: Combining multiple global memory accesses into one transaction.
  • PTX: Parallel Thread Execution; a low-level virtual machine and instruction set.
  • Occupancy: Ratio of active warps to the maximum possible warps on an SM.
  • Bank Conflict: When multiple threads in a warp access different addresses in the same shared memory bank.

AI_QUIZI_QUIZ. Why does CUDA use a 3D hierarchy (Grid/Block/Thread) instead of a simple 1D array of threads?

  • Answer: To naturally map to multidimensional data (images, volumes, matrices) and to optimize data locality within the hardware's Streaming Multiprocessors.
  1. What is the primary difference between SIMD and SIMT?

    • Answer: SIMD requires the programmer to explicitly pack data into vectors. SIMT allows threads to be written as scalar code, with the hardware handling the parallel execution and branching.
  2. If a kernel uses 64 registers per thread on a GPU with a limit of 65,536 registers per SM, what is the maximum number of threads that can run on that SM?

    • Answer: $65,536 / 64 = 1,024$ threads.
  3. How does __syncthreads() differ from a standard CPU barrier?

    • Answer: __syncthreads() only synchronizes threads within a single block, not the entire grid.

AI_STUDY_GUIDEI_STUDY_GUIDE*Key Objectives for Mastery:**

  1. Index Mapping: Practice converting 2D and 3D coordinates into linear memory addresses for various data layouts (Row-major vs. Column-major).
  2. Memory Optimization: Write a kernel that uses shared memory to perform a reduction (e.g., summing an array) and optimize it to eliminate bank conflicts.
  3. Profiling: Use nsys and ncu to identify the bottleneck in a slow kernel—is it "Compute Bound" (math limited) or "Memory Bound" (bandwidth limited)?
  4. Architecture Awareness: Research the differences between NVIDIA architectures (e.g., Pascal vs. Ampere) regarding shared memory size and L2 cache behavior.
  5. Modern C++ Integration: Experiment with thrust::device_vector and std::execution::par_unseq to see how high-level abstractions map to the CUDA primitives discussed here.

Source Materials

Study Introduction to CUDA and Modern C++ Programming on Lykke

This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.

Add the free extensionSign in to open every section

All sections of Introduction to CUDA and Modern C++ Programming · Browse more course pages