In this article
System memory broadens supported placement options, but generation speed and GPU capacity require a complete hardware and workload profile.
A server with 128 GB RAM can support CPU inference, supported CPU-GPU placement, data preparation and surrounding LLM services. System-memory capacity is not GPU VRAM and does not determine generation speed by itself. Describe the whole machine and test a specific role before deciding whether it needs an accelerator upgrade.
Record the complete configuration
Include CPU model and physical layout, available RAM, storage, GPU and VRAM, operating system and competing services. A nominal memory total does not show how much the application can use. Preserve the actual available resources and the intended operating window.
NVIDIA’s memory guidance distinguishes physical memory and mechanisms for shared access. A unified address space does not mean all memory regions offer equal performance or that every application automatically supports their use. The selected implementation determines which placement routes can actually be tested.
Compare realistic server roles
| Role | Contribution of RAM | Primary test |
|---|---|---|
| CPU inference | Holds a supported model | Response time on real inputs |
| CPU-GPU placement | Supports an available offload route | Transfers and peak memory |
| Document preparation | Buffers and parallel jobs | CPU, disk and working set |
| Retrieval service | Indexes and caches | Query latency under load |
These roles are not interchangeable proof of capability. A server useful for preprocessing may have unacceptable interactive generation delay. Conversely, a CPU or retrieval task can be valuable without consuming a large GPU at all.
Verify the selected engine
llama.cpp supports compatible CPU and hybrid CPU-GPU execution routes. It provides a practical experiment path, not a guarantee that every architecture fits because the server has substantial RAM. Record the exact model file, build, backend, context and placement settings.
Start with one short control request, then representative input and output lengths. If offloading is used, label the result as mixed placement and measure the complete response. Do not add RAM and VRAM together as if they described one physically uniform accelerator.
Measure latency, queues and the next step
Test a single task before adding the intended simultaneous workload. Record memory, CPU use, storage activity, waiting time and complete results. A successful load does not show whether the server meets an interactive deadline or remains usable for its other agreed tasks.
The evidence may support using the existing server, adding a GPU, changing the model or assigning a different pipeline stage. For an equipment owner or operations partner, that is a clearer decision than buying from a memory label alone.
Official references: docs.nvidia.com — understanding memory, github.com — llama.cpp.
Related guides: A bounded PEFT and LoRA experiment on your GPU, 256 GB RAM for RAG: when the index justifies the capacity.