An agent I was running called an enrichment API, got a response, and trusted it. The response was syntactically perfect — valid JSON, all required fields present, HTTP 200. The agent moved forward with the data. It made a decision based on that data. Then another decision. Then a write. When I finally traced back through the logs twelve hours later, I found the root: the API had returned a value that was off by a factor of one thousand — a price: 1000 when it should have been price: 0.001. The response satisfied every validation the agent had.

The cost wasn’t $0.001 worth of damage. It was three decisions, a write that triggered a refund process, a customer escalation, and twelve hours of debugging. $800 in incident cost to fix a data value that was wrong by $0.999. The failure mode is what surprised me: not an error, but a silent data corruption. And it’s the one thing retry logic can’t save you from.

This is validation debt: the cost you pay when you skip the check on a tool response, and the cascade multiplies it.

The incident

The enrichment API is a third-party service (though the pattern is the same for any tool call). It takes a product ID and returns enrichment metadata — category, price, stock status. The agent uses this to decide whether to recommend the product, set a sale price, and commit the recommendation.

The API contract is simple:

{
  "id": "string",
  "price": "number",
  "category": "string",
  "in_stock": "boolean",
  "last_updated": "ISO8601"
}

The agent’s code to call it was equally simple:

def enrich(product_id):
    r = requests.post("https://enrichment.service/v1/enrich", 
                      json={"id": product_id})
    r.raise_for_status()
    return r.json()

def process_product(agent, product_id):
    data = enrich(product_id)  # <-- trust the response
    
    # Step 1: Should we recommend?
    recommendation = agent.decide(f"Product {product_id} is in category {data['category']} "
                                  f"at price {data['price']}. Recommend?")
    
    # Step 2: Set price
    if recommendation == "yes":
        margin = 0.20
        markup = data['price'] * margin
        final_price = data['price'] + markup
    
    # Step 3: Write
    db.update(product_id, {"recommendation": recommendation, "final_price": final_price})

The agent called this flow for a batch of 10,000 products. One of them got a price value that was 1000× off — 1000.00 instead of 1.00. The response was a valid JSON number. No exception was thrown. The agent’s validation accepted it.

Here’s what happened:

  • Step 1: The agent reasons “price is $1000, category is mid-range… recommend? This is an expensive item, but the category suggests it should be, so yes.”
  • Step 2: The agent calculates the margin: 1000 * 0.20 = 200, so final_price = $1200.
  • Step 3: The agent writes the record with recommendation: yes, final_price: 1200.
  • Step 4 (human process): The pricing system flags this as an anomaly (product normally prices at $1.20 margin). A human reviews, sees it’s way off market, escalates it, and a refund is issued to the customer who purchased at $1200.

The bad data didn’t just waste a call. It triggered three subsequent decisions, each one building on the poisoned value, and finally an irreversible action (the write) that cascaded into a customer escalation. The cascade didn’t require the agent to be stupid — it required the agent to be blind, trusting the tool’s word because it came from an HTTP response.

Why the validation was skipped

The same reason most validation is skipped: the developer believed they knew the contract. The enrichment API’s docs promised price is a number. The JSON parser would reject malformed JSON. What more was there?

Three things:

  1. Range. A number is a number, but 1000 and 1.00 are both valid JSON numbers. The contract didn’t say price is between 0 and 100. The API had no range validation on its own output.
  2. Type inflation. The API implemented price as a database float, and floats can hold 1000.00 fine. But the intended range is 0–100, and once a float escapes that range, downstream code that assumes the range breaks silently.
  3. Cross-field consistency. Price and category should correlate. A mid-range product shouldn’t cost $1000. No tool checks this — it’s a human semantic invariant.

The validation that would have caught this cost two lines:

def enrich(product_id):
    r = requests.post("https://enrichment.service/v1/enrich", 
                      json={"id": product_id})
    r.raise_for_status()
    data = r.json()
    
    # Validate before trusting
    if not (0 <= data.get("price", -1) <= 100):
        raise ValueError(f"Price out of range: {data.get('price')}")
    
    return data

The agent would have caught the bad response, the tool call would have raised an exception, and the agent’s error-recovery would have kicked in — most likely by skipping this item or escalating it. Same as if the API had returned a 500. Cost: zero. The bad write never happened.

Why it cost so much: silent failures compound harder than loud ones

In a previous post, I showed how a loud failure — an error thrown by a tool — creates a retry cascade that can grow 75× worse when nested retries multiply. That’s expensive, but at least it’s visible. An alert fires. The error propagates fast enough to trip a spend ceiling.

Silent failures are worse because they don’t alert you to stop. The agent thinks it succeeded, so it keeps going.

Error (loud):     Tool throws 400
                  → Agent re-plans
                  → Tool throws 400 again (5 times)
                  → Agent finally gives up
                  → Result: attempt aborted, cost bounded by retry caps
                  
Bad data (silent): Tool returns 200 with price=1000
                  → Agent reasons on bad data
                  → Agent makes decision
                  → Agent makes decision based on that decision
                  → Agent writes state based on that chain
                  → Result: bad state committed, cost is cleanup + incident

The costs multiply differently:

  1. The immediate cost of the bad call: Negligible. One API call.
  2. The cost of reasoning on bad data: The agent re-reads the context (which now includes the bad value) and emits tokens discussing it. This happens N times as the agent reasons through the cascade. For an 8-step agent run, that’s 8 × “re-read bad data + emit reasoning” = 8 token-pairs.
  3. The cost of follow-on actions: If the agent does something based on the bad reasoning — writes a database record, sends a message, triggers a process — you now have human work to undo it. In this case: refund, customer escalation, investigation.
  4. The cost of debugging: Finding the root cause. Twelve hours of logs from 10,000 items, searching for the one that went wrong and tracing the cascade back to the API response.

The formula looks like this:

IN_PRICE, OUT_PRICE = 3.0 / 1e6, 15.0 / 1e6  # $/token, Sonnet-class

bad_call_cost      = 0  # one API call, negligible
reasoning_cost     = 8 * (6000 * IN_PRICE + 400 * OUT_PRICE)  # 8-step cascade, re-reading context
human_work_cost    = 300  # refund, escalation, customer outreach
debugging_cost     = 500  # 12 hours at fully-loaded cost

total_cost_per_item = bad_call_cost + reasoning_cost + human_work_cost + debugging_cost
# ~$800

items_affected = 1  # in this case, one
print(f"Total: ${total_cost_per_item * items_affected:.0f}")

The bad data itself isn’t the cost driver — it’s the cascade it triggers, and the human work to undo it. A one-line validation check would have cost zero (one string comparison per API call) and prevented $800 in downstream costs. The validation debt is the unpaid bill, and the interest is paid in cascading failures.

How validation debt scales to a fleet

This incident was a single agent running batch processing. When you scale to a fleet — thousands of agents calling the same tool — the calculus gets worse.

  1. Correlated failures: If the enrichment API returns bad data for a category of products, every agent in the fleet will cascade on it independently. What was one customer escalation becomes a thousand. The API returned bad data once, but N agents all built decisions on it in parallel.
  2. Harder to detect: With one agent, you find one bad record. With a thousand agents, you have a thousand bad records spread across your system before you notice the pattern. The debugging cost scales with fleet size.
  3. Silent spread: A tool that throws an error is caught by circuit breakers, shared budgets, and error-rate dashboards. A tool that returns bad data silently spreads through your fleet’s outputs — and if those outputs are inputs to other tools, the contamination multiplies as it travels.

The validation pattern that scales is to push it to the tool boundary, not inside each agent:

class ValidatedEnrichmentClient:
    """Wrapper that validates API responses before trusting them."""
    
    SCHEMA = {
        "price": {"type": "number", "min": 0, "max": 100},
        "category": {"type": "string"},
        "in_stock": {"type": "boolean"},
    }
    
    def enrich(self, product_id):
        r = requests.post("https://enrichment.service/v1/enrich", 
                          json={"id": product_id})
        r.raise_for_status()
        data = r.json()
        
        # Validate schema
        self.validate(data)
        
        # Check cross-field invariants
        if self._invalid_category_price_pair(data["category"], data["price"]):
            raise ValueError(f"Invalid category/price pair")
        
        return data
    
    def validate(self, data):
        for field, rules in self.SCHEMA.items():
            value = data.get(field)
            if value is None and field != "optional_field":
                raise ValueError(f"Missing field: {field}")
            if "min" in rules and value < rules["min"]:
                raise ValueError(f"{field} too low: {value}")
            if "max" in rules and value > rules["max"]:
                raise ValueError(f"{field} too high: {value}")

The wrapper becomes the single source of truth for “this data is safe to use,” and every agent that calls the tool gets the same guarantee. Validation happens once, at the boundary, not repeated inside each agent.

What I’d do

The one-line fix (a range check) would have prevented this. The rest prevents the class of it.

  1. Validate at the tool boundary. Every external tool call should have a schema validator — type, range, required/optional fields. Push this outside the agent logic, into a wrapper around the tool. One validation per tool call, not repeated in every agent.
  2. Make validation cheap and fast. A validator shouldn’t make a network call or call the LLM to check. It should be a schema check — types, ranges, regex, cardinality bounds. Under a millisecond per call.
  3. Fail loud on validation errors. If data doesn’t validate, raise an exception (not a warning, not a log line). The agent’s retry/error-recovery logic will catch it. This gives you the same error-handling machinery as any other tool failure — circuit breakers, budgets, escalation.
  4. Log the validation failure and the bad data. When a tool fails validation, log the raw response and the validation error. This is your trace for debugging the tool’s bug, not your agent’s cascade.
  5. Distinguish validation errors from transport errors. A 500 is a transport error and should trigger a retry. A validation error means the tool returned garbage — usually a symptom of a tool bug, not a transient fault. Retry transport errors; don’t retry validation errors (the next attempt will fail the same way).

In the fleet context:

  1. Share the validator across agents. Don’t write validation in every agent — write it once in a shared ValidatedToolClient and pass it to all agents. Changes to the schema (new field, range adjustment) happen in one place.
  2. Version your validators. As the tool’s contract evolves, your validator evolves with it. Use schema versioning so you can handle both old and new responses during a migration — and you can catch the tool’s silent contract breaks.
  3. Monitor validation failures. A tool that fails validation is signaling that its output is corrupt. Validation-error rate is a leading indicator of tool degradation — more sensitive than raw error rate, because it catches “responses that look fine but are wrong” before cascades happen.

The distinction from retry logic

This is the complement to fleet-retry-patterns and how agent failures cascade. That post was about bounding a visible failure — errors that get thrown and can be caught. This is about preventing an invisible failure — data that looks good but is corrupt.

  • Retry logic answers: “When a tool throws an error, how do we bound the blast radius?”
  • Validation answers: “How do we prevent a tool from poisoning agents with bad data that doesn’t throw an error?”

Retries are about recovery. Validation is about prevention. Both are necessary. A well-instrumented agent has:

  • Validation at the tool boundary (prevents bad data from entering)
  • Retry logic with circuit breakers (bounds the spread if an error does escape)
  • Cascade detection inside the agent (stops a contaminated run before it writes state)

Validation is the cheapest to implement and has the highest prevention leverage. Retry and cascade controls are the safety net when validation misses something.

The numbers here are a reconstruction: the $1000 price, the three-decision cascade, and the $800 total cost are a self-consistent model of a real incident, stated so you can swap in your own token costs and human work rates. The lessons — validate before trusting, push validation to the boundary, distinguish validation from transport errors — are tool and model-agnostic.