Why Split Prefill and Decode GPUs Lose Time in the Hand-Off, Not the Network

Engineer's view. This is a Node.js server doing a synchronous parse on every upload. Each parse is quick, but it blocks the one loop every connection shares, so every response slows. With a fixed connection pool, slower responses then mean waiting for a free connection. The decode server's hand-off work behaves the same way. TL;DR. When prompt... Weiterlesen: Why Split Prefill and Decode GPUs Lose Time in the Hand-Off, Not the …