https://loopandretry.github.io/tags/metrics/ Recent content in Metrics on Loop & Retry https://loopandretry.github.io/images/og-default.png https://loopandretry.github.io/images/og-default.png Hugo en-us Mon, 13 Jul 2026 18:10:00 -0400 https://loopandretry.github.io/posts/measuring-agent-failure-in-production/ Mon, 13 Jul 2026 18:10:00 -0400 https://loopandretry.github.io/posts/measuring-agent-failure-in-production/ Most agent failures don’t throw. The run returns a result, exit code zero, and the result is wrong — or it burns an hour and quietly gives up. If your monitoring only counts exceptions, you’re blind to the failures that actually cost you. A taxonomy of agent failure modes and the specific instrumentation that catches each one before your users or your bill do. https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/ Wed, 08 Jul 2026 01:30:00 -0400 https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/ A model grading your agent’s output is the only thing that scales for subjective quality — and it’s a biased instrument you’re reading as a ruler. Here are the biases that actually move scores (position, verbosity, self-preference, leniency), why raw agreement with a human hides them, and how to validate and harden a judge with code — including why you should be reporting Cohen’s kappa, not accuracy. https://loopandretry.github.io/posts/what-to-measure-when-your-agent-works/ Tue, 07 Jul 2026 21:00:00 -0400 https://loopandretry.github.io/posts/what-to-measure-when-your-agent-works/ “It works” is a demo result, not a measurement. An agent is a trajectory, not a function, and grading only the final answer throws away most of what decides whether it’s reliable. Here’s a five-layer scheme for what to measure — outcome, trajectory, cost, failure class, and stability under nondeterminism — with a small harness that computes it.