In this article
Async in a function name does not establish overlap; verify prerequisites and observe overlapping execution intervals.
Distinguish submission from execution
An asynchronous call may return control to the CPU quickly without showing that a copy and kernel execute simultaneously on the GPU. Function return time and device execution time have different boundaries. The CUDA Best Practices Guide describes prerequisites for overlapping transfers and computation, including device capabilities, host memory and streams. Start with a specific hypothesis: which transfer should overlap which computation, and why no required dependency prevents it. If a kernel consumes data from that unfinished transfer, moving calls to different streams does not remove the need for a correct data access order.
Verify prerequisites
Record GPU, CUDA and driver releases, device properties, host memory type and selected streams. Check documented pinned memory requirements for the transfer path being examined. Then list dependencies between buffers and operations, including possible default synchronization. Multiple streams do not automatically imply independent work. A library can insert waits, or the application can reuse a buffer prematurely. The ownership diagram should explain which operation reads and writes each memory region. An apparent acceleration with corrupted data is not successful overlap, even when the timeline looks persuasive and average duration becomes smaller.
Create a sequential control
The first variant performs a known amount of transfer and computation sequentially. The second performs the same amount using explicitly independent work segments. Preserve warmup, sizes, repeat count and result validation for both. Comparing a large kernel with a different algorithm would make any gain impossible to attribute to overlap. Measure total duration using identical completion boundaries. The CUDA Runtime Event API describes event recording and interval measurements; their meaning depends on where events sit relative to the examined work. An event in one stream must not arbitrarily be treated as completion of all device work in other streams.
Inspect timelines and remaining gaps
The principal artifact is a timeline of transfers, kernels and waits accompanied by launch configuration. If intervals do not overlap, investigate hardware support, memory type, dependencies and hidden synchronization in that order. If they overlap but total duration does not improve, examine memory contention and workload size: overlap can change individual operation durations. Do not add isolated timings and call the difference proven savings without measuring a common control interval. Check the CPU preparing subsequent data as well. Its delay may leave the GPU idle, creating gaps that resemble an inability to execute operations concurrently.
Bound the practical conclusion
Report which operations overlapped, which total interval changed and whether data remained correct. See LLM memory bandwidth and GPU PCIe bandwidth. One synthetic result does not promise acceleration for every model or production batch. An inference engine may use different sizes, streams and dependencies that need separate observation. Accept the optimization on a verified workload rather than the Async name or stream count in source code. Preserve versions and experiment limits alongside results, so a later run can reproduce the comparison’s meaning instead of merely repeating its command.