We Got 2x LLM Inference Speed With Three Kubernetes Settings

Author: DigitalOcean - Bewertung: 1x - Views:8

Scaling LLM inference on Kubernetes sounds straightforward, until you realize every pod needs 140GB of model weights, and your network isn't tuned to handle it.



In this video, I walk through the reference architecture we built at DigitalOcean for running vLLM on DOKS with Managed NFS, and the two problems that quietly cost you real money if you ignore them.



We cover why NFS beats object storage, baked images, and block volumes for shared model weights and the race condition that can silently lock your GPU nodes at half throughput even after you tune the network.



Here's what we covered:



→ Why Managed NFS is the right storage layer for multi-replica LLM inference

→ How jumbo frames + TCP buffer tuning gets you ~2x NFS throughput on the same hardware

→ The race condition between your network tuner and vLLM pods, and how a node taint fixes it

→ Live demo



The full reference architecture is open source — Terraform, Kubernetes manifests, everything. Link below.



Reference Architecture - https://github.com/digitalocean/scale-with-simplicity/tree/main/reference-architectures/vllm-nfs



[0:00] - Intro

[0:50] - Architecture overview

[2:03] - Why NFS for model storage

[4:03] - Network tuning deep dive

[6:15] - The race condition + taint fix

[7:54] - Demo

[9:33] - Wrap





🚀 Join the Developer Cloud:

https://cloud.digitalocean.com/registrations/new?utm_source=youtube&utm_medium=organic_video&utm_campaign=digitalocean&utm_content=



// STAY CONNECTED

🌏 Follow our blog for the latest updates: https://www.digitalocean.com/blog

🦈 Join our Developer Community on Discord: https://discord.com/invite/digitalocean

🐥 Follow us on X/Twitter: https://x.com/digitalocean

👩‍💻 We're Hiring! See open roles: http://grnh.se/aicoph1