24 seconds per iteration instead of 0.4. I paid for six hours of GPU compute and trained on CPU the entire time.

Failure 1 — CUDA silently fell back to CPU My training job launched on Vast.ai and ran to completion. Iteration time was 24 seconds instead of 0.4 seconds. CUDA had fallen back to CPU silently. PyTorch logged nothing. I had been billed for six hours of GPU compute while training on an unaccelerated CPU thread the entire time. Failure 2 — HF_HOME...