AgentInspect lets developers capture an AI agent’s step-by-step actions as JSONL logs, making it possible to verify that required tools ran – all without wiring OpenTelemetry into the stack.
In a quick test with a weather-checking example built on the Vercel AI SDK, AgentInspect flagged a “silent failure”: the final answer was correct, but the weather-tool call never happened. The library’s process-level check caught the problem that a plain output test would have missed.
Why the execution path matters
An AI agent’s response is the end product of a chain that can include multiple model invocations and external tool calls. If a prompt is supposed to fetch live data, the absence of that fetch changes the risk profile even when the text looks right. Answer-level tests let a bug that skips a crucial step slip into production unnoticed.
What AgentInspect does
- Lightweight recording – A TypeScript adapter hooks into the agent code and writes each model call, tool invocation, and their ordering to a line-delimited JSON (JSONL) file.
- Rule engine – Developers declare simple predicates such as “the weather tool must be called” or “the finance API must not be used.” The library evaluates these rules after each run.
- CLI inspection – Failed runs are saved as files that the built-in command-line tool can open for a human-readable replay of the execution trace.
Because the logs are local files, there’s no need for a tracing backend, network configuration, or a separate observability service.
How the test was set up
- Process checks – A rule verified that the weather-tool endpoint was invoked.
- Answer checks – A separate assertion compared the generated text to the expected indoor-sightseeing recommendation.
Two scenarios were run:
| Scenario | Answer check | Process check |
|---|---|---|
| Happy path (weather fetched) | Pass | Pass |
| Skipped weather (tool never called) | Pass | Fail |
The second run demonstrates the value of process checks: the answer looked fine, but the missing tool call signaled a hidden defect.
AgentInspect vs. Promptfoo
Promptfoo, a more established testing framework, captures traces via OpenTelemetry. That approach works well for teams that already ship telemetry data to a collector, but it adds configuration overhead and a dependency on a tracing infrastructure.
AgentInspect feels earlier in development, but it is very simple to set up.
Who should care
- CI pipelines – Adding a single step that runs AgentInspect and fails the build on rule violations turns silent bugs into hard stops.
- Debugging – The JSONL logs can be opened locally to replay the exact sequence of calls, saving time compared to digging through remote trace dashboards.
- Product teams – Knowing that a critical data fetch actually occurred before shipping a feature reduces the risk of downstream user complaints.
Bottom line: when an AI agent’s correctness hinges on the actions it takes, not just the final text, a lightweight recorder like AgentInspect can turn hidden process bugs into visible, testable failures—without the overhead of a full tracing stack.