In this article
Measure the complete batch, including startup and recovery. Peak GPU speed multiplied by available hours does not establish useful output.
Not every AI task needs an immediate answer. Archives, generated descriptions and other background work can run in batches. For owners with predictable idle GPU windows, this is worth testing when the task genuinely supports scheduling and a clear completion deadline.
Define the work unit
Specify whether completion means a document, record or another batch element. Preserve expected output format and quality checks. Generated tokens alone do not establish successful processing of a dataset.
vLLM treats offline inference separately from online serving. AWS Batch Transform illustrates a managed batch product with its own rules. Do not apply one product’s capabilities to arbitrary equipment; choose a supported execution route and task boundary first.
Agree the schedule
Record window start, environment-preparation time and when the GPU must return to its owner. Long model loading reduces useful work time below the calendar window. For an illustrative overnight batch, define input cutoff and acceptable unfinished remainder. Working at night needs a deadline and someone responsible for any extension.
Identify automatic and human actions. Required intervention when nobody is available can invalidate a schedule even if the GPU has enough capacity.
Increase batch size in stages
Start with a small representative sample and increase volume, preserving elapsed time, errors, RAM, VRAM and temporary storage. Do not simultaneously change the entire dataset and concurrency without a separate comparison.
Check quality at every stage. Skipped difficult documents can make higher speed reflect less useful work. Include typical and unfavorable inputs: average short-record time does not describe a late remainder of long documents near the window’s end.
Test restart and results
Saved state should distinguish completed from repeatable work. Otherwise an interruption may require the whole batch again. Stop and resume an agreed test, checking for missing and incorrectly duplicated records. Account separately for repeated external-system calls where applicable.
Define temporary-storage lifecycles. A fast local disk should not be the only unexplained location of results needed after shutdown. Test the final small remainder too: uneven records can leave a slow tail after most work finishes. Plan deadlines around the entire task, not the best speed in the middle of a uniform batch.
Discuss window economics
Compare completed-batch cost including startup, recovery and waiting. Do not multiply peak GPU speed by every available hour as though work never pauses.
Prepare with spot scenarios, NVMe and startup and result accounting.