https://loopandretry.github.io/tags/evals/ Recent content in Evals on Loop & Retry https://loopandretry.github.io/images/og-default.png https://loopandretry.github.io/images/og-default.png Hugo en-us Tue, 14 Jul 2026 05:02:00 -0400 https://loopandretry.github.io/posts/predicting-agent-failure-before-release/ Tue, 14 Jul 2026 05:02:00 -0400 https://loopandretry.github.io/posts/predicting-agent-failure-before-release/ A demo proves an agent can succeed once. It says almost nothing about how often it will fail under real load, real input distributions, and real adversarial garbage. The failures that cost you in production are predictable before release — but only if you test the things that actually shift between the demo and the deployment. Four pre-release signals that forecast production failure, and the ones that don’t. https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/ Wed, 08 Jul 2026 01:30:00 -0400 https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/ A model grading your agent’s output is the only thing that scales for subjective quality — and it’s a biased instrument you’re reading as a ruler. Here are the biases that actually move scores (position, verbosity, self-preference, leniency), why raw agreement with a human hides them, and how to validate and harden a judge with code — including why you should be reporting Cohen’s kappa, not accuracy. https://loopandretry.github.io/posts/what-to-measure-when-your-agent-works/ Tue, 07 Jul 2026 21:00:00 -0400 https://loopandretry.github.io/posts/what-to-measure-when-your-agent-works/ “It works” is a demo result, not a measurement. An agent is a trajectory, not a function, and grading only the final answer throws away most of what decides whether it’s reliable. Here’s a five-layer scheme for what to measure — outcome, trajectory, cost, failure class, and stability under nondeterminism — with a small harness that computes it. https://loopandretry.github.io/posts/loop-drift/ Tue, 07 Jul 2026 10:00:00 -0400 https://loopandretry.github.io/posts/loop-drift/ The worst agent failures don’t crash — they keep working. A postmortem on loop drift: an agent that stayed busy for 40 steps without getting closer to done, why the model’s own progress reports can’t catch it, and the external signals and evals that can.