Why My Prefix Cache Missed Identical Requests

I enabled prefix caching in mini-vllm-rs, my Rust LLM inference engine for macOS, and submitted two identical requests concurrently to a cold cache. Neither reused a single token. Both computed the same prompt. Why? My cache published reusable pages only when a request finished. Both requests arrived before either had published anything, so... Weiterlesen: Why My Prefix Cache Missed Identical Requests