🔧 I built an eval harness to prove an LLM worked. It proved the opposite.

I set out to have a language model classify integration failures. I built an evaluation harness to prove it worked. The harness proved it wasn't worth using.

Final architecture: deterministic rules do the classification and structured extraction. The LLM does exactly one thing, writing plain-English incident summaries. Cheap rules to sort, AI to...