Paradigm Shifts of Eval in the Age of LLM

Paradigm Shifts of Eval in the Age of LLMs

LLMs requires some subtle, conceptually simple, yet important changes in the way we think about evaluation

I’ve been building evaluation for ML systems throughout my career. As head of data science at Quora, we built eval for feed ranking, ads, content moderation, etc. My team at Waymo built eval for self-driving cars. Most recently, at our fintech startup was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.