How to Build a Post-Launch Eval Canary That Tells a Real LLM Regression From Sampling Noise

Is the model actually getting worse, or did I just get unlucky on a handful of prompts? That question is why threads like "is it just me or is it dumber today" keep recurring, and it is the question a post-launch eval canary has to answer with a number instead of a feeling. The reference implementation here is livenerf, a long-running,... Weiterlesen