From the free course 2.1.Intro to CUDA C++ on Lykke
Asynchronous Execution and Streams
Key concepts: CUDA Streams · CUDA Events · Asynchronous Execution · Pinned Memory · Blocking vs. Non-blocking
Covers techniques for overlapping computation and memory transfers to maximize GPU utilization using Streams and Events.
Asynchronous Execution and Streams
In the early days of GPGPU programming, the execution model was largely synchronous: the CPU would send a command to the GPU and wait for it to finish before proceeding. This "stop-and-wait" approach, while simple, left significant performance on the table. Modern high-performance computing demands that we treat the GPU not as a simple co-processor, but as an autonomous execution engine capable of running multiple tasks simultaneously while the CPU manages high-level logic.
Asynchronous Execution is the paradigm that allows for the overlapping of host computation, device computation, and data transfers. By decoupling the submission of a command from its completion, developers can hide the latency of the PCIe bus and keep the GPU's streaming multiprocessors (SMs) saturated.
AI_SVGI_SVG## The Mechanics of Asynchrony At its core, asynchronous execution relies on the fact that the CUDA driver and the GPU hardware possess independent engines for different tasks. Most modern NVIDIA GPUs feature:
- One or more Compute Engines: Responsible for executing kernels.
- Two or more Copy Engines: Dedicated hardware for moving data between host and device (H2D) and device to host (D2H).
Because these engines are independent, a GPU can simultaneously execute a kernel while transferring data for the next kernel and sending results from the previous kernel back to the CPU.
The CUDA Stream: The Unit of Concurrency
A CUDA Stream is a sequence of operations that execute in issue-order on the GPU. While operations within a single stream are guaranteed to execute sequentially, operations in different streams may execute concurrently or be interleaved by the hardware scheduler.
Definition: A CUDA Stream is a software abstraction representing a queue of GPU commands. Commands in different streams can overlap, provided the hardware has sufficient resources (e.g., available SMs or Copy Engines).
Pinned Memory (Page-Locked Memory)
To achieve true asynchronous data transfers, one must first understand the relationship between the CPU's RAM and the GPU. By default, host memory is pageable. The Operating System may move this memory to different physical locations or swap it to disk.
When you perform a standard cudaMemcpy, the driver must first copy the data from your pageable buffer into a "pinned" (page-locked) staging buffer, then initiate a Direct Memory Access (DMA) transfer to the GPU. This double-copy is a major bottleneck.
Pinned Memory is host memory that is locked into physical RAM. Because it cannot be swapped out, the GPU's DMA engine can access it directly without CPU intervention.
| Feature | Pageable Memory | Pinned Memory (cudaHostAlloc) |
|---|---|---|
| Allocation | malloc or new |
cudaHostAlloc or cudaMallocHost |
| Transfer Speed | Lower (requires staging) | Higher (direct DMA) |
| Asynchrony | Synchronous only | Supports cudaMemcpyAsync |
| OS Impact | Flexible, low impact | Reduces available system RAM |
| Best Use Case | Small, infrequent data | Large datasets, streaming, real-time |
Why Pinned Memory is Required for Async
The cudaMemcpyAsync function returns control to the CPU immediately. If the memory were pageable, the OS might move the data while the GPU is trying to read it, leading to memory corruption or system crashes. Pinned memory provides the stability required for the hardware to manage the transfer independently of the CPU's memory management unit.
CUDA Streams: Default vs. Non-Default
Every CUDA application has a Default Stream (also known as the Null Stream). If you do not specify a stream when launching a kernel or calling a memory copy, the operation is assigned to the default stream.
The Null Stream Trap
Historically, the default stream was a "synchronizing stream." Any operation submitted to the default stream would wait for all other streams to finish, and no other stream could start until the default stream operation was complete. While modern CUDA allows for a "per-thread default stream" (compiled with --default-stream per-thread), the legacy behavior remains a common pitfall for developers expecting concurrency.
| Stream Type | Concurrency Behavior | Use Case |
|---|---|---|
| Default (Null) Stream | Blocks other streams; synchronous with host. | Simple scripts, debugging, serial logic. |
| Non-blocking Stream | Created with cudaStreamNonBlocking. Does not sync with Null stream. |
High-performance pipelining. |
| Priority Stream | Created with cudaStreamCreateWithPriority. |
Latency-critical tasks (e.g., UI or real-time control). |
Pipelining: Overlapping Computation and Communication
The primary motivation for using multiple streams is Pipelining. By breaking a large dataset into smaller "chunks" and processing them in different streams, we can hide the cudaMemcpy latency behind the kernel execution time.
The "4-Way Overlap" Pattern
In a typical pipelined implementation, we aim for the following overlap:
- Stream 1: Copying Chunk $N+1$ to Device.
- Stream 2: Executing Kernel on Chunk $N$.
- Stream 3: Copying Results of Chunk $N-1$ to Host.
If the time taken for the kernel execution is greater than or equal to the time taken for memory transfers, the data movement effectively becomes "free" in terms of total wall-clock time.
// High-signal example: Pipelined execution using multiple streams
void executePipeline(float* h_data, int totalSize, int chunkSize) {
const int numStreams = 4;
cudaStream_t streams[numStreams];
for (int i = 0; i < numStreams; ++i) cudaStreamCreate(&streams[i]);
for (int i = 0; i < totalSize; i += chunkSize) {
int s = (i / chunkSize) % numStreams;
int size = min(chunkSize, totalSize - i);
// 1. Asynchronous H2D Copy
cudaMemcpyAsync(d_in[s], &h_data[i], size * sizeof(float),
cudaMemcpyHostToDevice, streams[s]);
// 2. Asynchronous Kernel Launch
processKernel<<<grid, block, 0, streams[s]>>>(d_in[s], d_out[s], size);
// 3. Asynchronous D2H Copy
cudaMemcpyAsync(&h_results[i], d_out[s], size * sizeof(float),
cudaMemcpyDeviceToHost, streams[s]);
}
// Synchronize all streams before returning
for (int i = 0; i < numStreams; ++i) {
cudaStreamSynchronize(streams[i]);
cudaStreamDestroy(streams[i]);
}
}
AI_DEMOI_DEMO## CUDA Events: Synchronization and Timing
As we move toward asynchronous execution, we lose the ability to use standard CPU timers (like std::chrono) to measure GPU performance, because the CPU timer will stop as soon as the command is issued, not when it is finished.
CUDA Events are markers that can be inserted into a stream. They serve two primary purposes:
- Timing: Measuring the elapsed time between two points in GPU execution.
- Synchronization: Coordinating dependencies between different streams.
Event-Based Timing
To time a kernel accurately, you record a "start" event before the launch and a "stop" event after. You then use cudaEventElapsedTime to calculate the duration in milliseconds. This is more accurate than CPU-side timing because it measures the actual hardware execution time.
Cross-Stream Dependencies
Sometimes, Stream B depends on a result produced by Stream A. Rather than synchronizing the entire device (which is expensive), we can use cudaStreamWaitEvent. This tells Stream B to pause until a specific event in Stream A has been recorded, allowing all other streams to continue unimpeded.
Synchronization Primitives
Managing asynchrony requires a nuanced understanding of when to wait. CUDA provides several levels of synchronization, ranging from "heavy" (blocking the whole system) to "light" (waiting for a specific task).
| Function | Scope | Impact |
|---|---|---|
cudaDeviceSynchronize() |
Global | Blocks the host until all issued CUDA calls in all streams are complete. |
cudaStreamSynchronize(stream) |
Stream-level | Blocks the host until all commands in the specified stream are complete. |
cudaEventSynchronize(event) |
Event-level | Blocks the host until the specified event is recorded by the GPU. |
cudaStreamWaitEvent(stream, event) |
Device-side | Non-blocking for host. The specified stream waits for the event, but the CPU continues. |
Key Insight: Always prefer the narrowest possible synchronization. Using
cudaDeviceSynchronize()in a high-performance loop is a common anti-pattern that destroys the benefits of using multiple streams.
Unified Memory and Asynchrony
With the introduction of Unified Memory (UM) and cudaMallocManaged, the lines between host and device memory have blurred. In a UM context, the CUDA driver handles data migration automatically via page faults.
However, asynchrony is still vital. By using cudaStreamAttachMemAsync, developers can associate specific managed memory regions with specific streams. This hints to the driver that the data should be migrated to the GPU in anticipation of a kernel launch in that stream, effectively pre-fetching data and reducing the latency of on-demand page faulting.
Common Pitfalls and Best Practices
1. The "Implicit Sync" Trap
Certain CUDA operations cause an implicit synchronization of the entire device, even if you are using streams. These include:
- Any memory allocation/deallocation on the device (
cudaMalloc,cudaFree). - Setting device flags or configurations.
- Memory copies to/from the "constant" memory space.
- The default (null) stream behavior (unless compiled with per-thread default stream).
2. Resource Contention
While you can create hundreds of streams, the hardware has a limited number of Hardware Work Queues (often referred to as HyperQ). If you exceed the number of hardware queues (typically 32 on modern architectures), the streams will be multiplexed, and you may lose the benefits of true concurrency.
3. Forgetting Pinned Memory
A common mistake is calling cudaMemcpyAsync on a pointer allocated with malloc. In this case, the call will behave synchronously, blocking the host until the transfer is complete, because the driver cannot safely perform a DMA transfer from pageable memory.
4. Stream Priorities
In complex applications (like a game engine or a self-driving car system), some tasks are more urgent than others. Use cudaStreamCreateWithPriority to ensure that latency-sensitive kernels (like sensor processing) are scheduled ahead of throughput-heavy tasks (like background logging) when SM resources are scarce.
AI_FLASHCARDSI_FLASHCARDS## Summary of Execution Flow
- Allocate host memory as pinned (
cudaHostAlloc). - Create multiple non-default streams (
cudaStreamCreate). - Issue asynchronous commands (
cudaMemcpyAsync,kernel<<<... , stream>>>) to different streams. - Use Events for fine-grained synchronization and timing.
- Synchronize only when necessary, and at the narrowest scope possible.
By mastering these concepts, a developer transforms the GPU from a simple math accelerator into a sophisticated, multi-tasking parallel processor capable of overlapping every stage of the data processing pipeline.
Source Materials
Study 2.1.Intro to CUDA C++ on Lykke
This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.
Add the free extensionSign in to open every sectionAll sections of 2.1.Intro to CUDA C++ · Browse more course pages