Before scoring a model, test the evaluator

One version of my evaluator passed a Kubernetes configuration whose selector contradicted its own pod labels. It also rejected a configuration just because an extra label had been added. Both results were scoring errors. If you are new to evals, start with What exactly are AI evals?. It explains the tasks, graders, and metrics used in this case... Weiterlesen: Before scoring a model, test the evaluator