First touch and NUMA: locating GPU host buffers

CPU affinity does not describe every memory page

Placing a host thread near a GPU is only part of a data-transfer experiment. The pages backing its host buffers may have been allocated elsewhere or before the thread moved. A process can therefore look well placed in a CPU-affinity report while its data follows a different memory path.

The Linux NUMA memory-policy guide explains policy scope and allocation behavior. In particular, changing a task policy does not automatically relocate pages already faulted in under earlier conditions. This makes allocation timing a first-class part of a reproducible host-buffer experiment.

Define a narrow buffer test

Choose one GPU, one host-buffer type, a fixed buffer size and a defined transfer direction. Record the CPU and memory-node constraints actually available to the process. Container or scheduler restrictions can limit a requested policy, so the intended configuration is not sufficient evidence of actual placement.

For an anonymous test buffer, distinguish reserving address space from populating the backing pages. Record which thread initializes the buffer and under which policy. Treat library-managed, pinned and file-backed buffers according to their actual allocation path rather than assuming every allocation follows one universal first-touch rule.

Observe allocation and transfer together

The CUDA best-practices guide treats host-device data movement and memory management as distinct performance concerns. Keep buffer allocation or registration outside the steady transfer interval when that is the question being tested. If startup cost matters too, report it as a separate interval instead of silently excluding it from every result.

Capture observed memory placement where the platform exposes it, the GPU’s device path and the transfer’s completion boundary. Preserve units and whether the reported bandwidth covers one direction or a larger sequence. A faster allocation method does not necessarily produce the fastest repeated transfer path.

Compare newly prepared conditions

An illustrative comparison can use two freshly prepared buffers with the same contents and size but different documented placement conditions. Keep transfer code, GPU, direction and concurrency unchanged. Reusing one old buffer after moving a thread would not isolate the intended placement difference because its existing pages may remain where they were.

Repeat conditions in a recorded order and monitor other host work. If the actual memory-node observations do not differ, the experiment has not established the intended local-versus-remote comparison. Report that limitation rather than assigning the measured difference to NUMA by assumption.

Connect the result to the serving process

After the small test, examine which buffers the engine actually allocates, which threads initialize them and whether their lifetimes cross worker changes. A microbenchmark result does not establish the entire model server’s behavior. Tokenization, storage and device-local computation may dominate the final request time.

The useful artifact is a placement-and-timing record with allocation policy, observed pages, buffer type, transfer interval and workload conditions. It explains a specific memory path without promising that CPU pinning alone fixes GPU throughput or that changing a process policy moves all existing data.

The record also lets another engineer reproduce the tested experiment after a platform or container configuration changes.