Spot GPU Instances: Checkpointing and Budgeting for Interruptions
Spot and preemptible GPUs can be a large discount on on-demand rates — with the catch that the provider may take the machine back at short notice. This guide covers resumable jobs, the real cost including restarts, and which workloads should never run on spot.
All figures in this guide are illustrative examples for education. Real GPU prices, cloud rates and throughput vary by provider, region, model and time. Confirm current numbers with a provider before budgeting. The point here is the method; plug in your own numbers.
On this page
The deal and the catch
Spot capacity is hardware the provider is not currently selling at full price. You can rent it at a discount that is often large, on the understanding that when a full-price customer wants it, you get a short warning and then the machine is gone. The discount is real; so is the eviction. Our pricing guide covers where spot sits next to on-demand and reserved pricing.
The mistake is to treat spot as “on-demand but cheaper”. It is a different product with a different failure mode, and a job that was not designed for interruption will eventually lose everything since its last save. Designed properly, spot is the best price-performance available to small teams; designed badly, it is the most expensive instance type there is, because you pay for the hours and keep none of the work.
Making a job resumable
A resumable job can be killed at any moment and restarted on a fresh machine with little loss. The ingredients:
- Frequent checkpoints to durable storage. Save model state, optimiser state, the data position and the random seed to object storage or a network disk, not to the instance’s local disk, which disappears with the instance.
- A checkpoint interval you have chosen deliberately. Too frequent and the saves slow the job; too rare and each interruption costs more. A common starting point is to save often enough that an interruption costs no more than five to ten percent of a typical uninterrupted run.
- Idempotent start-up. The job’s start script looks for the latest checkpoint and resumes from it; if none exists, it starts fresh. No human decision in the loop.
- A supervisor that relaunches. Something outside the instance notices the eviction and requests a new one, retrying across instance types or regions if the first choice is unavailable.
- Handling the warning. Most providers give a short notice before reclaiming the machine. Use it to save a final checkpoint and exit cleanly rather than being cut mid-write.
Budgeting for interruptions honestly
The spot price per hour is not the cost of the work. The cost is the spot price multiplied by the hours you actually pay for, including the hours lost to interruption and the overhead of saving and restarting. A simple model:
Effective cost = spot rate × (useful hours + hours lost to interruptions + restart overhead)
| Scenario (illustrative) | On-demand | Spot, well designed | Spot, no checkpoints |
|---|---|---|---|
| Rate per hour | $3.00 | $1.00 | $1.00 |
| Useful compute needed | 20 h | 20 h | 20 h |
| Interruptions during the run | 0 | 3 | 3 |
| Work lost per interruption | – | 0.5 h | all progress since start |
| Hours paid | 20 | ~22.5 | unbounded; may never finish |
| Cost | $60 | ~$22.50 | unknown, often more than $60 |
Interruption rates vary by provider, GPU type, region and time of day, and providers publish little about them. Measure your own: log every eviction for a month and you will know the rate for your actual configuration, which is the only number worth planning with.
Workloads that should not run on spot
- Online serving with latency commitments. An eviction mid-request is a user-visible failure. Spot can supplement serving capacity behind a load balancer, but the baseline should be on-demand or reserved.
- Jobs that cannot checkpoint. Some pipelines hold state in memory that is expensive or impossible to serialise. If the job cannot resume, it should not be interruptible.
- Short jobs with hard deadlines. If the result is due in two hours and the job takes ninety minutes, one eviction misses the deadline. The discount is not worth the risk.
- Multi-node training without elastic support. Losing one node of eight can stall the other seven. Unless the framework handles node loss, run the cluster on-demand or reserved.
Tactics that raise the odds
- Be flexible about instance type. Accepting any of several GPU types that can run your job widens the pool and shortens the wait after an eviction.
- Be flexible about region. Keep data in storage reachable from more than one region so a restart elsewhere does not require a re-upload.
- Avoid the obvious peaks. Interruptions tend to cluster when demand is high. Long jobs started at the quietest part of the week see fewer evictions.
- Use smaller instances in parallel. Several small spot machines working on independent shards lose less per eviction than one large one.
- Mix. A reserved or on-demand core plus spot overflow gives you a guaranteed minimum pace and cheap acceleration when capacity is available.
A pre-flight checklist
- Checkpoints go to durable storage and resume has been tested by killing the job on purpose.
- The supervisor relaunches without a human, and alerts if it fails to find capacity for longer than a threshold.
- The eviction warning handler saves and exits cleanly.
- The budget includes an interruption allowance based on your measured rate, not the provider’s marketing.
- The deadline tolerates at least two restarts.
If every item is ticked, spot is likely to be the cheapest way to run the job. If any is not, the discount is a trap, and on-demand is the cheaper choice in practice.