https://loopandretry.github.io/tags/latency/ Recent content in Latency on Loop & Retry https://loopandretry.github.io/images/og-default.png https://loopandretry.github.io/images/og-default.png Hugo en-us Tue, 14 Jul 2026 05:00:00 -0400 https://loopandretry.github.io/posts/cost-beyond-tokens/ Tue, 14 Jul 2026 05:00:00 -0400 https://loopandretry.github.io/posts/cost-beyond-tokens/ Everyone budgets the token bill because the provider hands you an invoice for it. But an agent in production spends across five other axes that never show up on that invoice — wall-clock latency, orchestration, tool-call fees, human review, and idle polling — and for a lot of workloads the tokens are the smallest line. A model that sums all six so you can see which one you’re actually paying. https://loopandretry.github.io/posts/cheap-first-smart-later/ Mon, 13 Jul 2026 20:30:00 -0400 https://loopandretry.github.io/posts/cheap-first-smart-later/ Most requests to your agent are easy, and you’re paying frontier-model prices for all of them anyway. A routing cascade — try the cheap model, escalate on a measurable confidence signal — cuts spend without touching output quality, if you get the escalation trigger right. Here’s the pattern, where it breaks, and the arithmetic on when it’s worth building. https://loopandretry.github.io/posts/your-agents-p99-is-a-different-animal/ Mon, 13 Jul 2026 18:00:00 -0400 https://loopandretry.github.io/posts/your-agents-p99-is-a-different-animal/ Average latency is the number you demo and the number nobody experiences. A multi-step agent is a sum of random variables, so its total time is dominated by the tail of each step — and the more steps you add, the more certain it becomes that at least one of them is slow. Here’s the model, why the p99 of the whole is worse than the p99 of the parts, and the two levers that actually move it. https://loopandretry.github.io/posts/when-not-to-build-an-agent/ Wed, 08 Jul 2026 02:00:00 -0400 https://loopandretry.github.io/posts/when-not-to-build-an-agent/ An agent is an LLM that controls its own control flow — and that autonomy has a price you pay on every run: quadratic token cost, serial latency, and a failure surface you can’t unit-test. Most tasks people reach for an agent on are a fixed pipeline wearing a costume. Here’s the decision checklist I use, the arithmetic on what the agent tax actually costs, and the same task built both ways so you can see the difference.