DigitalOcean: How to Keep Training Through a Spot GPU Reclaim

I ran a full fine-tune of Qwen3-8B on an AMD Instinct™ MI350X spot GPU Droplet on DigitalOcean. Mid-run, I fired a simulated reclamation notice and timed the checkpoint, upload, replacement provisioning, restore, and resume.

The 98.3 GB checkpoint reached durable object storage in 322.6 seconds, or 5.4 minutes. A replacement Droplet resumed at step 14,625 with zero steps redone.

The video covers:

• What changes when you run training on reclaimable GPU capacity
• Which workloads belong on spot GPUs
• What a complete training checkpoint must contain
• Every measured stage from the notice to resumed training
• Capacity fallback, orchestration failure, and GPU nondeterminism
• How the checkpoint and recovery path scale to a 70B model
• MI350X spot pricing versus reserved pricing on the day of the run
• The checkpointing setup I would use for a production training job

This was one measured drill. I simulated only the trigger because a real reclaim cannot be scheduled.

The roughly 15-minute recovery time is the sum of individually measured stages, not one uninterrupted stopwatch run. DigitalOcean Spot GPU Droplets are in public preview.

Full methodology and code: https://github.com/bnarasimha21/amd-mi350x-spot-drill

Chapters
0:00 AMD Instinct™ Spot GPUs
0:35 What changes when the GPU is temporary?
2:03 Which workloads should run on spot GPUs?
2:38 Fine-tuning Qwen3-8B on an MI350X
3:33 What must a training checkpoint contain?
4:17 The measured reclamation drill
5:40 Capacity fallback, script failure, and GPU nondeterminism
6:30 Scaling the checkpoint path to 70B
7:33 Spot pricing versus reserved pricing
8:37 The spot GPU training playbook

🚀 Join the Developer Cloud:
https://cloud.digitalocean.com/registrations/new?utm_source=youtube&utm_medium=organic_video&utm_campaign=digitalocean&utm_content=HIJOeVPkirg

// STAY CONNECTED
🌏 Follow our blog for the latest updates: https://www.digitalocean.com/blog
🦈 Join our Developer Community on Discord: https://discord.com/invite/digitalocean
🐥 Follow us on X/Twitter: https://x.com/digitalocean
👩‍💻 We're Hiring! See open roles: http://grnh.se/aicoph1