Stop running fresh model benchmarks and start watching your agent try to cancel a subscription. The gap between those tw

Stop running fresh model benchmarks and start watching your agent try to cancel a subscription. The gap between those two activities is where production systems die. A single-turn test can tell you if a response sounds pleasant. It cannot tell you whether the agent just refunded the wrong customer, looped fourteen times against a calendar API, or decided to skip the fraud check entirely. Text is the least dangerous thing an agent produces. The real risks hide in the tools it touches, the data it mutates, and the moments when it should have asked for help but kept going.

Why Text Benchmarks Fail in Production

High scores on standard benchmarks have become a misleading form of comfort. An agent that writes elegant prose might still be an operational hazard. When your system books appointments, edits database records, or files support tickets, the generated text is only the visible surface of the workflow. Underneath, the agent is making concrete decisions about which endpoint to hit, what payload to send, and when to stop. It can top a reading-comprehension leaderboard while costing you money by double-booking resources, mutating the wrong row, or leaking sensitive state into a log file. You need to verify the mechanics of the work, not just the polish of the output. If an agent can score well on an offline QA test and still fail your workflow by looping or misusing a tool, your evaluation is looking at the wrong signals.

Mapping the Five Dependencies

The team at Van Data Team starts every evaluation by mapping five specific control points. This changes the question entirely. You stop asking if one model is smarter than another. You start asking if the agent can actually finish a production task under your real constraints.

Business outcomes. Define what "done" means in dollars and customer impact. A task is not complete because the agent emitted a summary. It is complete when the inventory record is accurate, the appointment is confirmed, and the customer received a valid tracking number.

Mutable state. Know exactly what the agent is allowed to change. Which tables, which statuses, which account flags? If the agent can issue refunds, reschedule jobs, or update billing addresses, you need to inventory every field it touches.

Tool permissions. Be explicit about which API endpoints and functions are in scope. An agent with access to a search tool, a write tool, and a notification tool will mix them up if the boundaries are fuzzy. Map each permission to a specific operational need.

Failure recovery. Decide what happens when the calendar API times out, returns a 500, or hands back malformed JSON. The agent should not panic, hallucinate a success message, or retry forever. It needs a clear fallback path.

Human review gates. Identify the moments where a person must sign off before the agent proceeds. This is not a sign of weakness in the automation. It is a safety valve for high-impact changes and a source of ground-truth labels for your rubrics.

What a Real Evaluation Plan Looks Like

Once the dependencies are mapped, you need an evaluation plan that matches the messiness of production. Slide-deck metrics will not help you here.

Build test sets from real production failures, not from synthetic question banks. If your agent failed last Tuesday by confusing two similar SKUs, that exact confusion should be a permanent test case. Your evaluation suite should grow every time an incident teaches you something new.

Write rubrics that define successful completion in operational terms. Vague criteria like "helpful" or "accurate" are useless. A useful rubric states that a refund task is only successful if the original payment ID was referenced, the amount matched the request, a confirmation email was queued, and the transaction ID was logged.

Define trace specs for tool calls and retries. You need observability into what the agent planned, what it actually called, how many times it retried, and whether the retry strategy was appropriate. A trace without tool-level granularity is just a pretty story.

Set policies for when to alert a human. The agent should know its own boundaries. If a request exceeds a dollar threshold, references a VIP account, or encounters a state it has never seen before, it should escalate rather than guess.

Install release gates to block bad model upgrades. A new model is only an upgrade if it improves your specific outcomes. If it hallucinates tool arguments more often, increases latency, or introduces new safety risks, it does not ship. The gate keeps production stable even when the base model vendor ships a new version.

Runtime Grading: Watching the Agent Work

Anthropic has been pushing the industry to move beyond offline tests toward runtime grading. Instead of judging a transcript after the fact, runtime grading lets a system judge the agent's work while the task is still in flight. This creates a chance to catch errors before they harden into real problems.

Adding a grader costs tokens and latency. You cannot afford to grade every tiny step. The placement of each grader is a design decision. Put them where the mistakes are expensive. The most valuable checkpoints sit just before committing a state change to a database, just before capturing a payment, and just before sending a message to a customer. These are the moments where a bad decision becomes an irreversible action.

Watch out for a specific blind spot. If the same model performs the work and also grades the work, it may miss the same mistakes. The reasoning that produced an error can easily rationalize that error away during review. For high-impact tasks, keep human review inside the loop. Let people validate the grader's own judgment, especially when money or customer trust is on the line.

The goal here is operational control. Connect your incident data, your task rubrics, and your runtime traces into one feedback cycle. Evaluate the whole path: the plan, the tool use, the recovery behavior, and the final result. Use offline tests to catch known, reproducible errors before release. Use runtime traces to find the new failures you did not anticipate. Use human review to discover where your rubrics are naive and need tightening.

So ask yourself: where would you put a runtime grader in your workflow? Before a tool call, after a tool call, or only before a risky change? Most teams start too broad, grading everything, and then grind to a halt under the cost. Start narrow. Pick the one action that would hurt the most if it went wrong. Put a grader there first.

Start With One Expensive Mistake

Operational evaluation is not a research exercise. It is a way to sleep better once the agent is live. You do not need a perfect framework on day one. You need a single, well-defined workflow, a rubric written in plain business terms, and a grader placed at the exact moment where an error becomes expensive. Get that right, and you have a foundation you can actually trust.

If you want to dig deeper into agent evaluation and runtime grading with a community of practitioners, you can find the GyaanSetu learning community at https://t.me/GyaanSetuAi.