How to Keep an AI Agent From Taking Dangerous or Unauthorized Actions

The answer is tool-scope minimization plus mandatory confirmation gates on any irreversible or high-blast-radius operation. Give the agent only the permissions it needs for the current task, never more, and force a human checkpoint before any action that cannot be undone or that affects more than one record at a time.

I am Mahmoud Zalt, an independent senior AI systems architect with 16+ years building production software since 2010. I founded Sista AI, where a year of running autonomous agents in production has taught me exactly how much damage an over-permissioned agent can do, and I now consult exclusively on production AI systems through my AI Agent Development practice. I have designed agent permission models for customer-support bots, autonomous coding agents, and internal ops tools. The failure mode is almost always the same: too many permissions granted upfront, no confirmation layer, and no audit trail when something goes wrong. This article gives you the practical framework I use, including the exact decision criteria for when a human must be in the loop.

Why AI Agents Are a Different Risk Profile Than APIs

A traditional API call is deterministic. You wrote the code, you know what it does, and it does exactly that. An agent is a reasoning loop: the model selects which tool to call, constructs the arguments, and decides whether to continue. That introduces three new failure modes that permissions must address.

• Prompt injection. Malicious content in retrieved documents or user messages can redirect the agent's tool selections. A customer-service agent that reads emails can be instructed by a crafted email to forward sensitive records to an attacker.
• Runaway chaining. Agents that can call tools in loops can trigger unintended cascades. An agent that can create calendar events and send emails can, if poorly scoped, spam every contact in a CRM.
• Hallucinated arguments. The model fills in tool arguments from context. A 'delete record' tool called with a hallucinated ID deletes the wrong record. There is no diff to review.

These are not hypothetical. I have seen all three in production engagements. The mitigation for all three is the same architectural posture: narrow permissions, mandatory confirmation, and full observability. The model's intelligence is not a substitute for a permission boundary.

Least-Privilege Tool Scopes: The Foundation

Every tool you expose to an agent is a capability you are granting to whatever the model decides to do next, including under adversarial conditions. The principle is identical to Unix file permissions: grant only the rights required for the current task, and grant them for the shortest time possible.

How to scope tools in practice

When defining tools, whether via OpenAI function calling, Anthropic tool use, or an MCP server, structure them at three tiers:

A tool named update_customer that accepts any field is worse than three tools: update_customer_email, update_customer_address, and update_customer_status. Narrow surface area means a misfired call does narrow damage. This feels verbose but it is the right call. In MCP terms, each tool definition should carry a description field that explicitly states its blast radius so the model can reason about when confirmation is appropriate.

Scoped credentials per agent role

Never give an agent a database connection with write access to tables it only needs to read. Use a dedicated service account or API key scoped to the minimum required operations. Rotate it. If the agent is compromised via prompt injection, the attacker inherits only that credential's permissions, not your entire data store.

Confirmation Gates: When and How to Pause the Agent

A confirmation gate is a synchronous interrupt in the agent loop. Before executing a tool in the high-blast-radius tier, the agent must surface a plain-language summary of what it is about to do and wait for a human to approve, modify, or reject it. This is not a UX nicety. It is a hard architectural requirement for any irreversible action.

The two criteria that require a gate

I use two independent criteria. If either is true, the action requires a confirmation gate:

1. Irreversibility. The action cannot be undone or recovery is expensive. Sending an email, deleting a row without soft-delete, charging a card, deploying to production, posting to a public channel.
2. Blast radius greater than one. The action affects more than a single record or a single user. Bulk updates, mass sends, anything parameterized by a wildcard or a list.

What the gate must show the human

A yes/no prompt is not enough. The gate must surface: the exact tool name, the exact arguments as the model constructed them, the number of records or users affected, and a plain-language explanation of what will change and what cannot be undone. Here is what a confirmation payload looks like in a real system I built:

Action: send_email
To: 3,847 contacts (segment: 'trial_expired_last_30d')
Subject: 'Your trial has ended'
Body: [preview first 200 chars]
Irreversible: yes
Estimated cost: $0.38 (SendGrid)

Approve / Edit / Reject

The human must be able to edit the arguments before approving. An agent that constructs a query slightly wrong, but whose confirmation gate only shows approve/reject, will get approved because the human trusts the model. Show the actual values.

Async vs synchronous gates

For long-running agents, synchronous blocking is impractical. Use a task-queue pattern: the agent writes a pending action to a queue with status awaiting_approval, emits a notification (Slack, email, webhook), and pauses. A human reviews asynchronously and transitions the status to approved or rejected. The agent resumes on the next poll cycle. This pattern scales to multi-step workflows where several high-blast-radius actions need sequential approval without blocking a thread.

Drawing the Human-in-the-Loop Boundary Correctly

The most common mistake I see is treating human-in-the-loop as a spectrum where more automation equals more trust. It is not. It is a binary classification per action type, made at design time, not at runtime.

The decision matrix

The bottom-right cell is not a confirmation gate problem. It is an architectural problem. If an agent can send an irreversible action to many targets in one call, that tool should not exist in the agent's tool set. You redesign the tool to accept a single target, and the agent must call it once per target, making the blast radius visible and enumerable in the confirmation gate.

Where teams get this wrong

The failure pattern I see repeatedly: a team builds the agent with a 'dry run' mode and treats that as equivalent to a confirmation gate. It is not. Dry run only catches what the model simulates. A real gate intercepts the actual call with the actual arguments. Another failure: gating only on the first occurrence. An agent in a loop can execute a confirmed action, then autonomously repeat it on new inputs without re-confirmation. Gate every execution of a high-blast-radius tool, not just the first one per session.

Observability and Audit Trails: You Need Both

Guardrails you cannot observe are guardrails you cannot trust. Every tool call the agent makes, whether it is a read or a write, whether it is approved or rejected, must be logged with enough context to reconstruct exactly what happened and why.

What to log per tool call

• Timestamp and session ID
• Tool name and full argument payload (redact PII if required by compliance, but log the structure)
• The model's reasoning trace or the prior message that led to this call
• Confirmation status: auto-executed, awaiting approval, approved by [user ID], rejected by [user ID]
• Result or error from the tool
• Latency and token cost for the step

This log is your post-incident forensics tool. When something goes wrong, and it will, you need to answer: what did the model see, what did it decide, who approved it, and what was the exact payload. Without this, you are debugging a black box.

Anomaly detection on top of logs

For production systems, parse the logs and alert on: tool call volume above a rolling baseline (runaway loop), arguments containing known-bad patterns (a delete call with no WHERE clause equivalent, a send call with unusually large recipient counts), and confirmation gates that are being approved in under five seconds (rubber-stamping, not real review). These signals surface behavioral drift before it becomes a production incident.

Prompt Injection Mitigations for Tool-Calling Agents

If your agent retrieves external content and uses it in its context window, that content is an attack surface. A document, email, web page, or database row can contain instructions that redirect the agent's behavior. This is prompt injection, and it is the primary security concern for retrieval-augmented agents.

Practical mitigations in production

1. Privilege separation between retrieval and action. The agent that reads external content should not be the same agent that executes write tools. A retrieval agent summarizes and passes structured output to an action agent. The action agent does not see raw user-controlled text.
2. Tool call validation layer. Before any tool call is dispatched, a lightweight validation layer checks the arguments against an allow-list of shapes. A delete call that arrives with an argument constructed from retrieved text (rather than from a known internal ID) is flagged and held for review.
3. System prompt anchoring. Explicitly instruct the model in the system prompt that no instruction in retrieved content, user messages, or tool results can modify its tool-use permissions, override confirmation requirements, or change its operating role. This does not fully prevent injection but raises the bar significantly.
4. Input sanitization for structured fields. Any field that will be passed directly as a tool argument (an ID, an email address, a SQL fragment) should be validated against a strict schema before the agent sees it. Do not let raw retrieval output flow into tool argument slots.

None of these is a complete solution alone. Layered together, they reduce the attack surface to the point where injection requires a sophisticated, targeted attempt, not a casual one-liner in a document.

Cost and Rate Controls as a Guardrail Category

Runaway agents are not just a data integrity problem. They are a cost problem. An agent in an unintended loop calling an LLM with a large context window can burn through a meaningful budget in minutes. Treat cost controls as a first-class guardrail, not an afterthought.

Controls to implement

• Max steps per run. Every agent invocation gets a hard ceiling on the number of tool calls it can make. Ten is a reasonable default for most task-oriented agents. If the task requires more, that is a signal the task scope is too large, not that the ceiling is too low.
• Token budget per run. Track cumulative input and output tokens across the run. When the run exceeds the budget, the agent must stop, emit a summary of what it completed, and return control to the caller.
• Rate limiting per user or tenant. In multi-tenant systems, one user cannot exhaust the agent quota for all users. Implement per-user rate limits at the API gateway level, not inside the agent.
• External API call limits. If the agent calls third-party APIs (SendGrid, Stripe, Twilio), enforce call limits per run. A bug that causes the agent to retry an API call in a loop should hit a circuit breaker, not your monthly invoice.

I typically implement these as a wrapper around the agent's run loop that tracks state and throws a BudgetExceededException. This is separate from the confirmation gate system but equally non-negotiable in production.

Frequently Asked Questions

How do I prevent an AI agent from deleting data it should not touch?

Do not give the agent a delete tool unless deletion is explicitly in scope for the task. If deletion is in scope, scope it to a single record per call (never bulk), log every call, and require a synchronous human confirmation gate before execution. Use soft deletes where the data model allows it, so accidental deletions are recoverable. Never give the agent's service account DROP or TRUNCATE privileges at the database level regardless of what the tool definition says.

What is the difference between a guardrail and a confirmation gate?

A guardrail is a broad category of controls: input validation, output filtering, tool-scope limits, rate limits, prompt injection mitigations. A confirmation gate is one specific guardrail: a synchronous or asynchronous human-approval checkpoint on a high-blast-radius or irreversible tool call. Guardrails can run without human involvement. A confirmation gate by definition requires a human decision.

Can I use an AI model to review the agent's actions instead of a human?

Yes, and this is called an LLM judge or a critic agent. It is a useful layer for flagging anomalies and filtering obvious bad calls before they reach a human. However, it does not replace human confirmation for irreversible actions. A critic agent can be injected with the same malicious content that fooled the original agent. Use LLM judges for high-volume, low-blast-radius screening, and keep humans in the loop for anything that cannot be undone.

How do I handle prompt injection in agents that read emails or documents?

Separate the retrieval step from the action step architecturally. Validate all structured arguments before they reach a tool call. Anchor the system prompt with explicit instructions that no external content can override tool permissions. Sanitize known-bad patterns in retrieved text before it enters the context. None of these fully eliminates the risk, but layered together they raise the bar high enough to stop opportunistic attacks.

What is the minimum viable guardrail setup for an early-stage AI agent?

At minimum: (1) tool-scope limitation with no high-blast-radius tools available by default, (2) a hard max-steps ceiling on every run, (3) full logging of every tool call with its arguments, and (4) a confirmation gate for any tool that sends a message, modifies more than one record, or cannot be reversed. This set takes roughly a day to implement and prevents the majority of production incidents I have seen.

How do multi-agent systems change the permission model?

In multi-agent systems, each agent should have its own minimal permission set based on its role. An orchestrator agent that delegates to subagents should not itself hold the credentials or tool access that the subagents hold. Confirmation gates should be placed at the orchestrator level so a human reviews the orchestrated plan, not every individual subagent step. Treat inter-agent communication as an untrusted channel: a subagent receiving instructions from an orchestrator should validate those instructions against its own permission policy, not blindly execute them.

Build Agents That Cannot Hurt You

The question is not whether your agent will encounter an edge case that tries to make it do something dangerous. It will. The question is whether your architecture makes that edge case survivable. Least-privilege tool scopes, mandatory confirmation gates on irreversible actions, full observability, and a clear human-in-the-loop boundary are not advanced features. They are table stakes for any agent that operates on real data or real users.

I have been designing and building these systems since before 'agentic AI' was a mainstream term. If you are building an AI agent for production and want an architecture that will not embarrass you in a post-incident review, I do this work through my AI Agent Development practice. Start with my work at /about and /projects, or reach out directly at /contact.

Talk to me about building AI agents that are safe by design.