Light Just Cut KV Cache Memory Traffic to 1/16th


The bottleneck in long-context LLM inference isn't compute. It's memory bandwidth.

Every decode step in a Transformer scans the entire KV cache to generate a single token. That's O(n) memory reads for context length n, every single step. No matter...