Field notes on building LLM agents that survive production. Concrete code, real failure modes, numbers over adjectives — context, tools, evals, cost, and the ways agents break.
Testing multi-agent crews: forcing split-brain, goal drift, and context drift to happen in CI
Coordination failures don’t show up in a test suite because the test run happens to schedule agents politely. Split-brain needs a specific interleaving, goal drift needs a specific chain of lossy handoffs, and context drift needs a correction that lands mid-run. All three can be forced deterministically: a scripted scheduler that controls who reads and writes when, a handoff chain with an assertion on the constraint ledger instead of on the output, and a mid-run policy change with a stale-holder check. Concrete harness code, what each test asserts, and why you should not assert on model output at all.
Tool design for reliable multi-agent crews: preventing coordination failures at the tool boundary
Every fix this cluster has proposed for split-brain, goal drift, and context drift is a discipline — something an agent has to remember to do. Agents are bad at remembering and excellent at doing what a schema requires, which makes the tool boundary the cheapest place to prevent a coordination failure: a tool that cannot express a stale write cannot cause a split-brain. Concrete signatures for compare-and-set state, constraint-carrying handoffs, and staleness-aware reads — plus the three tool shapes that manufacture these failures on purpose.
Observability for multi-agent systems: instrumenting for split-brain, goal drift, and context drift
Every coordination failure in a multi-agent system is a diff between two agents — and per-agent monitoring cannot express a diff. You can have complete, exception-free telemetry on every agent in a crew and still be structurally blind to split-brain, goal drift, and context drift, because each one lives in the comparison your telemetry has no join key for. What to actually instrument for each of the three, which probes are free and which cost a model call, and how to sample a crew for belief reconciliation without paying O(n) extra calls at every checkpoint.
Context drift: when a multi-agent crew's shared beliefs about the goal go stale
A cache that’s never invalidated doesn’t know it’s stopped being correct — it just keeps serving the value it read at write time. Give a multi-agent crew a shared objective instead of a shared cache and you get the identical failure with none of the tooling that usually catches it: every agent read the goal once, at spawn, and nothing tells it when that copy goes stale. This is multi-agent context drift — distinct from goal drift (the target itself moving) and split-brain (the crew’s facts about state disagreeing) — a concrete policy-escalation failure, the mechanism, and what to instrument before a customer hears a promise nobody’s operating on anymore.
Goal drift: how a multi-agent crew ends up solving a different problem
Loop drift is one agent staying busy without getting closer to done. Hand the same failure to a crew and every agent stays busy, produces defensible work, and the team’s collective objective still ends up somewhere your original request never sent it — a genuine multi-agent goal drift failure mode, not a mistake in any single agent’s reasoning. A concrete feature-build failure, the four things that make a goal decay one handoff at a time, and what to measure so a ‘done’ check compares against the original ask instead of the nearest upstream spec.
Split-brain: when multi-agent memory consensus quietly fails
In distributed systems, split-brain is what happens when a partition convinces two halves of a cluster that each is the sole leader, and both start accepting writes. Multi-agent systems build the identical topology every time more than one agent reads and writes a ‘shared’ memory store with no lock, no leader, and no version check — and the failure that follows is a genuine multi-agent memory consensus problem, not a bug in any one agent. The mechanism, a concrete incident-response failure, and what to instrument before it costs you a duplicate action.
Context engineering: what fits and what gets dropped
“Context engineering” is getting used as a rebrand for prompt writing, and that undersells it into uselessness. The actual discipline is deciding, every single turn, what earns a place in a bounded window and what gets evicted — and living with the fact that every eviction is a bet you’re making about the future of a run you haven’t finished yet. Here’s the physical constraint underneath the buzzword, the packing decision that’s actually hard, what it costs to get it wrong, and how to test whether your packing policy works.
How indirect prompt injection actually happens
Every indirect prompt injection follows the same four-stage pipeline — placement, retrieval, interpretation, execution — and that pipeline is the actual answer to ‘how does this happen,’ not a list of clever attack strings. Here’s the mechanism, where it hides in ordinary agent traffic you already run, and the defenses that target each stage instead of hoping the model classifies intent correctly.
What an MCP server actually is, and the tool-design mistakes that break it
MCP gives you a standard wire format for connecting an agent to tools — it doesn’t make the tools on the other end safe to call. Most MCP servers are thin wrappers around an existing REST API, which quietly imports every assumption that API made about having a human developer as its client. Here’s what MCP actually is, how it differs from the API you already know, and the specific tool-design mistakes that turn a working MCP server into a flaky agent.
What Idempotent Actually Means: Why Retries Are Safe (and When They Aren't)
Idempotent means doing something twice has the same effect as doing it once. This is essential for APIs, agents, and any system where retries can happen. Here’s why, and how to get it right.