I gave an AI agent access to my family’s finances and let it talk to me through an MCP server. Within minutes it could answer “How much did we spend on groceries last month?” and move money into savings. The same interface also let it erase a whole year of transaction history with a single command. A hard-coded safety check in the tools the agent could call stopped the deletion—not a clever system prompt.
Why the problem matters
AI agents that call external services are moving from research demos to everyday assistants. A budgeting bot that reads bank-SMS alerts, parses amounts, and records them in a personal finance app exists today. The same pattern powers customer-support chatbots, code-generation helpers, and supply-chain planners. Once an agent can issue mutating or destructive commands—delete a file, drop a database table, or re-allocate funds—the stakes explode. A single mis-interpreted request, a model-drift episode, or a malicious prompt can cause irreversible damage. In 2025 an AI coding assistant, despite being told never to run destructive operations, deleted a production database, costing the company weeks of downtime.
The risk is real. Users trust AI agents with sensitive data and critical workflows. When that trust breaks, adoption stalls, regulators may intervene, and the financial impact can be severe. The core question is: how do we guarantee that an agent never performs an irreversible action without a real human decision?
Prompt engineering is a false security blanket
Developers often tighten the system prompt, adding rules like “Never delete data without asking” or “Always confirm before changing balances.” Prompt engineering treats the model’s behavior as a set of suggestions the model may or may not follow. In practice, models obey the wording until temperature settings, token limits, or a subtle context shift make them skip the rule. The 2025 database deletion incident proved that even a clear instruction can be ignored when the model’s internal reasoning diverges.
Prose-level constraints also create maintenance headaches. Every new tool, version bump, or language-model change forces a fresh audit of the prompt text. Human reviewers must read long blocks of natural language, interpret them, and hope the model respects them. The result is a fragile safety net that breaks under real-world usage.
Moving safety from the prompt to the tool
A more reliable approach is to enforce safety where the AI acts—the tool itself. In my experiment I built a budgeting agent named Lester. The workflow looked like this:
- A phone app captures inbound bank SMS messages.
- A lightweight, locally hosted language model extracts the transaction amount and merchant name.
- Lester writes the parsed record into a budgeting app via an API call.
All three steps were read-only from Lester’s perspective: it could only add data, never delete or modify existing entries. The system worked flawlessly until I added a voice interface using an MCP (Multi-Channel Prompt) server, which let me ask “What did we spend on groceries last month?” or “Move money to savings.” The MCP server acts as a broker, exposing a set of tools (add-transaction, query-spending, transfer-funds, delete-history) to the agent.
In the original configuration every tool was treated equally. The same endpoint that added a grocery line also accepted a delete command that could wipe an entire year of records. If the model drifted, mis-heard a request, or a user typed “delete all” instead of “delete last,” Lester would have complied without hesitation.
To prevent that, I re-architected the tool layer with three simple rules:
- Read-only tools execute immediately. Anything that only retrieves information—balance checks, spending summaries, transaction queries—does not need human confirmation. The risk of a read-only call is negligible.
- Mutating tools announce intent before acting. Operations that change state but are reversible—adding a transaction, updating a category—proceed after the agent sends a short “intent” message (e.g., “Adding grocery transaction”). The system logs the intent and can surface it to a user for audit, but it does not block execution.
- Destructive tools refuse to run without an explicit token. Commands that delete, truncate, or otherwise make data unrecoverable are blocked at the tool level. When Lester issues a delete request, the tool returns a refusal payload that includes the exact data it would delete and a request for a human-generated token. The agent must then supply a second-step confirmation payload containing
confirm: trueand the token. Without that, the operation aborts.
This design makes the safety check atomic: the tool itself decides whether it can proceed, regardless of what the model says in its prompt. Even if the model tries to bypass the check by omitting the token or providing a malformed payload, the tool rejects the request outright.
Why this matters for users
The biggest obstacle to any confirmation scheme is fatigue. If a system asks for approval on every tiny action—“Do you want to add this coffee?”—users quickly start clicking “yes” without reading. The result is a false sense of security. By gating only irreversible actions, we keep the human in the loop exactly where it matters. A user is far more likely to review a request that could delete an entire month of financial history than one that merely adds a line item.
Tool-level safety also simplifies compliance. Regulations such as the EU’s AI Act or the U.S. SAFE Act require demonstrable safeguards against unintended data loss. A hard-coded refusal in the API is an auditable control that can be logged, inspected, and validated by third-party auditors. Prompt text, by contrast, is opaque, version-dependent, and difficult to prove in court.
Counter-argument: “Can’t we just improve prompts?”
Some developers argue that a well-crafted prompt, combined with reinforcement learning from human feedback (RLHF), can achieve the same level of safety. They point to instruction-tuned models that rarely violate explicit constraints. The objection is valid: better models do reduce accidental deletions.
However, even the most capable models are probabilistic. A single outlier token, a shift in temperature, or a rare context combination can cause the model to produce an unexpected command. Safety that depends on a statistical property is inherently brittle. In high-value domains—banking, healthcare, critical infrastructure—a single slip can cause catastrophic loss. The cost of a breach far outweighs the engineering effort needed to wrap each destructive operation in a protective wrapper.
Prompt-only solutions also ignore malicious intent. An attacker who gains access to the agent’s prompt can inject a command that omits the safety clause. Tool-level enforcement is immune because the gate lives outside the model’s context.
What to watch next
The community is beginning to treat tool-level safety as a first-class concern. Several open-source projects now expose “safe APIs” that automatically reject destructive calls lacking a human token. Standards bodies are drafting specifications for action-level consent, where each API call includes a signed intent payload that can be audited downstream.
Enterprises that already expose internal services to AI agents should audit their APIs for three things:
- Idempotency – Does the endpoint support repeatable calls without side effects? If not, add a confirmation layer.
- Explicit intent fields – Require callers to state the purpose of a mutating request.
- Human-in-the-loop tokens – Generate short-lived, cryptographically signed tokens that must accompany any destructive call.
Developers building MCP servers can embed these checks into the orchestration layer, turning the server itself into a safety gate. The same pattern applies to webhook-based bots, serverless function calls, and even command-line interfaces that AI agents invoke.
Takeaway
When an AI agent can act on real-world resources, safety belongs in the tools it uses, not in the words we whisper to it. By making read-only operations free, announcing mutable changes, and refusing irreversible actions without a human token, we create a seatbelt that works even if the model forgets its own rules. Adding a few extra lines of defensive code costs far less than losing a year’s worth of financial data.