In this article
Compare interruption consequences as well as hourly rates. Spot is most useful to evaluate for workloads that preserve results and resume on available capacity.
Interruptible GPU capacity may look cheaper than ordinary rental. The outcome depends on work lost at interruption and how quickly replacement capacity becomes available. For equipment owners and GPU-pool funders, this concerns architecture and operating budgets as much as choosing a tariff.
Understand the two modes
AWS Spot Instances use spare capacity for flexible, interruptible work. Google Spot VMs have their own provisioning and interruption conditions. Check exact rules, GPU availability and pricing for the intended region and configuration. One cloud’s conditions do not apply universally to every offer labelled spot.
On-demand rental still needs recovery from failures. What differs is the resource-provisioning model, not the existence of universal protection against application interruption.
Choose a suitable workload
Background processing of independent documents differs from an interactive session with a waiting user. First determine whether work can be divided into completed units with results saved between them. For an illustrative archive, record each document’s ID, processing status and result location. A restarted worker should recognize completed items instead of restarting the entire set.
This is an application requirement rather than a property conferred by the rental label. Preserve the expected completion deadline and identify which interruptions the workflow can tolerate before judging the apparent hourly saving.
Rehearse interruption and recovery
Stop an agreed test workload within a controlled window and restore it on an available configuration. Measure obtaining the machine, loading the model and returning the application to readiness. Account separately for recomputation and data transfer.
A low hourly rate does not establish a low cost per completed batch if substantial work is repeated. Also preserve the case where no suitable replacement is available. For a deadline-sensitive job, this may matter more than average recovery time under successful provisioning.
Compare a mixed scheme
A persistent base resource with extra interruptible capacity is one architecture to test for suitable tasks. Show useful completions, waiting, restarts and total elapsed time in the pilot record. Maximum GPU utilization is insufficient: recomputing lost work also creates utilization.
Use a bounded experiment budget to discover where flexibility justifies additional complexity without promising savings or returns. Distinguish resource interruption, application error and rejected invalid input in the batch log. They require different repeat actions; merging them into one error rate makes costs and recovery choices harder to explain.
Assessment notes
Owners can propose resources with defined availability windows. Describe them accurately: a privately owned machine is not automatically equivalent to cloud spot capacity, although its breaks also require planning.
Related reading: Hosting batch inference: using GPUs on a schedule; Reserved GPUs and flexible capacity: what are you committing to?.