https://loopandretry.github.io/pillars/evals-testing/ Recent content in Evals-Testing on Loop & Retry https://loopandretry.github.io/images/og-default.png https://loopandretry.github.io/images/og-default.png Hugo en-us Wed, 08 Jul 2026 01:30:00 -0400 https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/ Wed, 08 Jul 2026 01:30:00 -0400 https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/ A model grading your agent’s output is the only thing that scales for subjective quality — and it’s a biased instrument you’re reading as a ruler. Here are the biases that actually move scores (position, verbosity, self-preference, leniency), why raw agreement with a human hides them, and how to validate and harden a judge with code — including why you should be reporting Cohen’s kappa, not accuracy. https://loopandretry.github.io/posts/what-to-measure-when-your-agent-works/ Tue, 07 Jul 2026 21:00:00 -0400 https://loopandretry.github.io/posts/what-to-measure-when-your-agent-works/ “It works” is a demo result, not a measurement. An agent is a trajectory, not a function, and grading only the final answer throws away most of what decides whether it’s reliable. Here’s a five-layer scheme for what to measure — outcome, trajectory, cost, failure class, and stability under nondeterminism — with a small harness that computes it.