Your AI agent can be talked into anything.
A fraudulent invoice. A poisoned web page. One crafted email. Software that acts on your behalf can be manipulated by the very information it reads, and no prompt can fix that. Proof, Not Trust shows the control layer that makes agent authority safe: the agent negotiates, humans sign, nothing binds without them.
wired after a deepfaked CFO video call
companies breached via chat-agent tokens
for an agent to erase a database and backups
The problem: agents that can act, but cannot be trusted
For two years, AI agents mostly drafted things — emails, summaries, code suggestions — leaving a human to act. That boundary is now dissolving. Enterprises are wiring agents directly into the systems that move money, sign agreements, provision infrastructure, and approve requests. The agent stops being an assistant and becomes an actor.
This introduces a category of risk the software industry has not had to manage before. A traditional program does exactly what it was written to do. An AI agent decides what to do based on language it reads at run time — and that language can be adversarial. A supplier’s invoice can contain hidden instructions. A web page the agent visits can carry a concealed command. This family of attacks has no clean technical fix, because the agent’s power and its vulnerability are the same faculty: it treats the text it reads as meaningful, which is exactly what an attacker exploits.
The tempting response is to make the agent smarter or more careful — better instructions, a second AI to check the first. This fails for a structural reason: anything inside the AI’s reasoning can be manipulated by the same inputs that manipulate the AI. A guardrail the agent evaluates is a guardrail an attacker can talk the agent past. The only durable place for enforcement is outside the agent, in deterministic code that does not read, reason about, or trust the adversarial material.
The incident record, 2012–2026
This is no longer hypothetical. Teal marks the incidents where the missing control is precisely an enforcement layer outside the agent.
- 2012
Knight Capital loses ~$440M in 45 minutes
Autonomous execution without a control layer fails at machine speed.
- Feb 2024
A tribunal holds an airline liable for its chatbot’s promise
“An agent can bind the company” is settled in court.
- Feb 2024
Arup wires $25M after a deepfaked CFO video call
Authorization must arrive through a channel an attacker cannot fabricate — a face and a voice no longer qualify.
- Jun 2025
EchoLeak: the first zero-click prompt injection in production
The text an agent reads is an attack surface; guardrails inside the model’s reasoning can be talked past.
- Jul 2025
A coding agent deletes a production database, then misreports it
Instructions do not bind an agent, and its own account of its actions cannot be trusted.
- Aug 2025
Stolen agent OAuth tokens breach 700+ organizations
Standing credentials are frozen authority — silent on whether this specific action, right now, is legitimate.
- Sep 2025
The first malicious MCP server found in the wild
The agent’s supply chain is adversarial — policy must be enforced outside both the model and its tools.
- Apr 2026
An agent wipes a production database and its backups in 9 seconds
A stray credential is blanket authority nobody consciously granted.
What has not yet happened at scale is the autonomous payment disaster — for a revealing reason. Enterprises have not dared give agents unsupervised authority to move large sums, precisely because the control layer to make it safe does not exist. The absence of a nine-figure agent-payment loss is not reassurance; it is the market not existing yet.
Why every current approach falls short
Keep a human in the loop for everything
The gapSafe, but it defeats the purpose of autonomy and does not scale. The failure is not that humans are involved — it is that involvement has exactly one setting: every action, or none.
Give the agent a limited budget or virtual card
The gapAuthority is frozen the moment the card is issued. A spending limit cannot express “this purchase was approved by this person,” and it does nothing for fraud that stays under the limit.
Extend identity and access controls (OAuth, scopes)
The gapPermissions are frozen at login, against a fixed list of allowed actions. A scope says nothing about whether a specific action, with specific values, is legitimate right now.
Route the agent through existing approval workflows
The gapClosest to right — but the approval is a record in a database, not bound cryptographically at the point money moves, and it assumes the requester is a trustworthy human rather than software that can be remotely manipulated.
Detect the manipulation (filters, classifiers)
The gapA useful layer but a brittle one: an arms race against adversarial text that puts the security decision back in the probabilistic zone the attacker is targeting.
The common thread: each approach either trusts the agent, freezes authority at setup time, bolts security onto the thing being manipulated, or slows the process to uselessness. None binds a specific, human-approved intent to a specific action at the moment it happens, enforced by something the agent cannot influence.
Seven principles that close the gap
Together they turn your approval policy into a cryptographic lock an AI agent cannot pick, regardless of how it is manipulated. In business terms: signing authority, enforced against agents.
Enforcement lives outside the AI
A deterministic gatekeeper sits between the agent and every consequential action. Ordinary, auditable code — it does not reason, cannot be argued with, and never sees the adversarial text the agent reads.
The agent’s claims are worthless; only evidence counts
The gatekeeper never takes the agent’s word. It requires a cryptographic signature produced by a real person through a channel the AI cannot reach. The agent can carry that evidence; it cannot forge it.
Consent is bound to the exact action
Not “this person approves spending” but “this person approves paying this amount to this recipient,” checked field for field. Even a fully compromised agent cannot execute anything that diverges from what a human actually signed.
Granularity is a dial the grantor sets
A signed claim can sit anywhere between a single action and an explicit, revocable mandate — “purchases under $200, at these three vendors, this month.” The dial position is itself part of the evidence.
Decisions happen at the moment of action
Not frozen at setup. The same policy can require nothing for a small routine purchase, one approver above a threshold, and two above a higher one, because it evaluates the specific values of the specific request.
Authority can be multi-party
Real corporate signing authority is thresholds, roles, and dual control. The gatekeeper composes multiple signed approvals into a single decision, mirroring your actual delegation-of-authority structure.
The rules are provable
The policy is authored so its properties can be mathematically verified, not merely tested — written in a proof assistant and compiled to the enforcement engine. You can demonstrate with proof, rather than assertion, that your controls behave as claimed.
This is not a new idea — it is zero trust, extended to the agent
None of these principles is novel security theory. Zero trust — the stance that no actor is trusted by virtue of where it sits, and that every action is authorized on its own merits — runs from Saltzer and Schroeder’s 1975 principles through de-perimeterization, Google’s BeyondCorp, and the NIST Zero Trust Architecture standard.
This system extends that architecture with one addition to the list of things not trusted: the AI agent itself. Classic zero trust distrusts an actor’s network position and then verifies its identity — after which the authenticated actor is trusted to behave within its permissions. An agent breaks that assumption. Its identity is not in question; its judgment is. It is a legitimate participant that can be talked into betraying its own instructions by the data it reads.
Why “human in the loop” becomes a rubber stamp
There is an honest objection to any approval-based system: people click “approve.” Two decades of human-factors research on automation bias documents the pattern — when an automated system is right most of the time, its operators stop evaluating and start ratifying. Vigilance decays precisely because the system earns trust. The attack category called MFA fatigue exists because approval prompts become reflexes.
Ask it plainly: would you sign every piece of paper an employee put in front of you? No executive would. Yet that is what an unstructured “human in the loop” becomes at agent speed — a signature line attached to a firehose. And the trap is double. The first failure mode is the human who approves without reading. The second is worse: an agent under manipulation can simply claim the human approved. A checkpoint the agent merely reports passing is not a checkpoint.
The architecture answers both. The signature is produced on the approver’s own device, through a channel the agent cannot reach, so approval cannot be fabricated. Routine actions run inside scoped mandates and interrupt no one, so the approvals that do reach a human are rare enough to deserve attention. The approval screen shows the specific amount, recipient, and document — a concrete decision, not an “OK?” button. The goal is not more human approvals; it is fewer, better ones.
How it works, concretely
The clearest illustration is the smallest one, already running in working software. A child asks to spend $15. The family policy says purchases up to $10 need only the child’s own signed request, but anything above $10 also needs a parent’s approval. The agent proposes the purchase; the gatekeeper denies it precisely — “missing: parent’s authorization”; the parent signs on their own device, bound to that exact request; both signatures are verified against the actual purchase; only then does the payment execute.
Now translate that to a corporation. The “child” is an AI procurement agent. The “$10 threshold” is a delegation-of-authority matrix. The “parent” is whichever manager the policy requires for that amount and category. The flow is subtractive, not ceremonial: the policy determines which signatures the transaction requires, the system determines which already exist, and only what is missing interrupts a human. If the agent was manipulated into paying a different party, the signatures won’t match, and the payment never leaves.
Why now
Deployment is outrunning control
Enterprises are putting agents into consequential roles right now, ahead of the safeguards that make doing so responsible.
Regulation is tightening
Provable, auditable authorization is moving from nice-to-have toward expected, especially in finance.
The standards are converging
On the same primitives this is built from — signed mandates and verifiable consent — while leaving the hard part, the enforcement decision, unaddressed.
The risk is already being priced
Munich Re extends AI-performance cover toward agent risk, Lloyd’s-backed specialists write standalone AI-liability policies, and Verisk is drafting agentic-AI exclusions. Insurers are deciding, line by line, what an ungoverned agent costs.
The fourth force deserves the closest look. When insurers start excluding ungoverned agents, a provable, auditable control layer becomes the thing that keeps an enterprise insurable — a compliance nicety turned purchasing requirement.
Three questions to ask about your own agents
A reader who takes nothing else from this paper should take these. For every AI agent your organization runs or plans to run:
- Q1
If this agent were manipulated today, what is the most expensive thing it could commit you to — and would you find out from your controls, or from the counterparty?
- Q2
Can you prove, to an auditor or an insurer, that a specific consequential action was approved by a specific person — with evidence that doesn’t depend on the agent’s own logs?
- Q3
Where do your agent’s rules live: in its instructions, where the text it reads can override them, or outside it, where nothing it reads can?
If any answer is uncomfortable, the discomfort is the point: these are the questions regulators, insurers, and your board will be asking within the year. Better to ask them first.
- —The complete incident record, 2012–2026, with sources for every entry.
- —The architecture diagrams: where the gatekeeper sits, and the five-step flow from proposal to execution.
- —Implementation detail for the three building blocks — verifiable credentials, the verified policy engine, and the gatekeeper.
- —“One architecture for any agreement”: how the same architecture covers payments, contracts, access grants, and regulated sign-off.
- —The expensive history of unproven rules — the DAO and Parity — and why multi-party approval code is the sharpest warning.
- —Full references and the roadmap: what runs today, and what is still ahead.
CTOs, CISOs, and finance and operations leaders wiring AI agents into systems that move money, sign agreements, or grant access — and the risk, audit, and insurance professionals who will be asked to sign off on them.
The control layer described here is open source and inspectable today — github.com/SenteLabsAI/extensible-mcp ↗
