AI agent governance checklist: 50 controls before agents go to production

Most AI governance documents were written for chatbots: what employees may paste into ChatGPT, which use cases need review, who signs off. Agents break those assumptions. An agent doesn't just answer. It calls tools, reads and writes data, makes dozens of model calls per task, and does all of it without a person reading each step.

This checklist is for the team that has to say yes to putting agents in production: platform, security, and whoever owns the AI budget. It's organised in ten areas, with what to check, why it matters, and how to tell whether you have it. Use it as a review before launch, and again every quarter.

At the end there's a table showing which controls a gateway can enforce centrally and which have to live in your agents or your process. No single tool covers all fifty, and anyone who says theirs does is selling.

1. Inventory: know what's running

You can't govern agents you don't know about. Most organisations that do an inventory for the first time find more agents than they expected, many of them a prototype someone left running.

• Every agent is registered with a name, an owner (a person, not a team alias) and a one-line purpose.
• Each agent has a risk tier, for example: read-only internal, writes to internal systems, touches customers or money.
• You know which models each agent uses, including fallbacks.
• You know which tools and data sources each agent can reach.
• Agents without an owner are switched off, not left running "until someone checks".

How to tell: pick any model call from last week's logs. Can you say which agent made it and who owns that agent? If not, start here.

2. Identity and access: every agent is its own principal

Shared keys are the most common governance gap. If five agents use one API key, you can't limit, audit or revoke one of them without affecting the other four.

• Each agent has its own credentials: its own API key for model calls, its own service accounts for tools.
• Agent credentials can't manage the account. The key an agent uses to call models can't create keys, change billing or read other agents' secrets.
• Admin access is separate and scarce. Keys that can change configuration are held by people or by a small, named set of operator agents.
• Credentials are stored in a secrets manager, never in prompts, code or agent memory.
• You can revoke one agent in under five minutes without touching anything else.
• Credentials rotate on a schedule and whenever an owner leaves.

How to tell: revoke one agent's key in staging. Does exactly one agent stop?

3. Data: what may go in, what may come out

Agents move data in both directions: into prompts, out to tools, back from models. Every hop is a place where personal data or secrets can leak.

• Each agent has a data classification: which classes of data it may send to a model (public, internal, personal, regulated).
• Personal data is redacted or blocked before it reaches a provider, unless the agent's purpose requires it and the provider is approved for it.
• Secrets are blocked. API keys, tokens and private keys in a prompt are refused outright, not redacted and forwarded.
• Responses are checked too. A model can echo back data it was given or produce content you don't want shown to users.
• Prompt injection is screened on content the agent didn't write: web pages, emails, documents, tool results.
• You know where prompts are stored, for how long, and by whom: your logs, your gateway, each provider.
• Data residency is documented for each provider the agent can reach.

How to tell: send a test prompt containing a fake card number and a fake API key through each agent. What reaches the provider?

4. Tools: what agents may do

Tools are where agents stop being text generators and start taking actions. Most serious agent incidents involve a tool, not a model.

• Tools are allowlisted. An agent can only reach tools and MCP servers that have been explicitly approved, not anything with a URL.
• Each tool has a scope: read-only or write, and which systems it touches.
• Write actions are least-privilege. An agent that drafts refunds doesn't hold credentials that can issue them.
• High-impact actions need a person: payments, deletions, sending external email, changing permissions. The agent proposes; a human approves.
• Tool calls are logged: which tool, which agent, which run, success or failure, and how long it took.
• Tool results are treated as untrusted input, because a tool that reads the web or an inbox can carry injected instructions back into the agent.
• New MCP servers go through review before they're added, like any other third-party integration.

How to tell: list every tool each production agent called last month. Was every one of them on an approved list?

5. Spend and rate limits: cap the blast radius

An agent stuck in a loop can make thousands of model calls in an hour. Alerts tell you after the money is gone; limits stop the next request.

• Each agent has a hard spend cap, daily and monthly, enforced before calls reach a provider.
• There's an organisation-wide cap above the per-agent ones.
• Rate limits apply per agent, in requests and tokens per minute.
• Agents handle limit errors by stopping cleanly, not by retrying in a tight loop.
• Cost is visible per run, so you can see what one task cost, not just the monthly total.
• Someone reviews the most expensive runs every week.

How to tell: set an agent's daily cap to $1 in staging and run it in a loop. Does it stop at $1, and does it stop gracefully?

6. Models: which ones, and how they change

• There's a list of approved models per risk tier. A customer-facing agent doesn't silently switch to an unreviewed model.
• Fallback models are approved too. Failover to another provider is good for uptime, but only to models you'd have chosen anyway.
• Model retirements are tracked. Providers retire models on their own schedule; know what each agent is moved to.
• Model changes are tested on a fixed set of tasks before they reach production.

How to tell: for one agent, can you name the exact model that served each of its calls yesterday?

7. Audit and observability: the record

When something goes wrong, the first question is "what exactly did the agent do?". The answer has to exist before the incident, not be reconstructed after.

• Every model call is logged with agent, run, model, tokens, cost, status and time.
• Calls are grouped into runs, so you can replay one task from first call to last.
• Logs avoid storing content you don't need. Metadata is usually enough for audit; store prompts only where you have a reason and a retention period.
• Logs are retained for as long as your policy or regulator requires, and can be exported.
• Configuration changes are recorded: who changed a key, a cap, an allowlist or a guardrail, and when.

How to tell: pick a run from last week at random. Can you list every model and tool call it made, in order, in under ten minutes?

8. Incident response: the off switch

• There's a kill switch per agent (revoke its key) and one for all agents (an organisation-level stop), and on-call knows how to use both.
• Agent incidents have a runbook: runaway spend, data leak, harmful output, unauthorised action.
• Alerts fire on anomalies: spend spikes, error spikes, an agent calling a tool it never called before.
• You've rehearsed it. Someone has actually pulled the switch in staging this quarter.

9. Change management: agents are software

• Prompts, tools and model choices are versioned like code, with review.
• Changes to a production agent go through staging with the same limits and guardrails as production.
• Each agent is re-reviewed when its tools, data access or risk tier changes, and at least yearly.

10. Vendors and compliance

• Each model provider has been reviewed for data use (training, retention), residency and security attestations.
• Data processing agreements are in place with every provider and gateway that handles personal data.
• Your controls map to the framework you report against, whether that's internal policy, the NIST AI Risk Management Framework, ISO/IEC 42001, the EU AI Act or Singapore's Model AI Governance Framework. The checklist above is the operational evidence those frameworks ask for.

Where each control lives

Some controls can be enforced in one place for every agent; some have to live in the agent's code or in your process. A gateway that every model and tool call passes through is the natural place for the first kind. Here's how that splits, and where Leanroute stands today:

Control	Best enforced in	Leanroute today
Separate credentials per agent	Gateway + secrets mgr	Yes: one API key per agent, revocable on its own
Agent keys can't manage the account	Gateway	Yes: runtime keys can't manage keys, billing or provider keys
PII and secret redaction or blocking	Gateway	Yes: guardrail library, on prompts and responses
Prompt injection screening	Gateway + agent	Partial: prompts are screened; MCP tool results aren't yet
MCP server allowlist	Gateway	Yes: per organisation, with discovery for agents
Tool allowlist per agent	Gateway	Not yet: the allowlist is per organisation today
Human approval for high-impact actions	Agent / workflow	No: build this in your agent or workflow tool
Spend caps and rate limits per agent	Gateway	Yes: daily and monthly caps, requests and tokens per minute, per key
Cost per run	Gateway	Yes: agent runs, in the dashboard and the API
Approved models, failover within them	Gateway	Yes: provider allow and block lists; failover off per request
Model call log, grouped by run	Gateway	Yes: metadata only, no prompt content stored
Tool call log	Gateway	Yes: tool names, outcomes and timings; never arguments or results
Audit trail of configuration changes	Gateway	Not yet: no exportable admin audit log
Kill switch per agent and for everything	Gateway	Yes: revoke a key, or set the organisation cap to zero
Prompt and model versioning, staging	Your repo and CI	Outside the gateway
SSO for the people who manage it	Gateway	Not yet
SOC 2 report	Vendor	Not yet

The pattern: identity, data rules, tool access, limits and the record of what happened belong at a chokepoint every call passes through. Approval workflows, versioning and review belong in your process. A good setup uses both, and is honest about which is which.

Using this checklist

Don't try to tick all fifty before the first agent ships. Start with the ones that limit damage, because they're cheap and they cover mistakes in everything else:

1. One key per agent (section 2).
2. Hard spend caps per agent (section 5).
3. A tool allowlist (section 4).
4. PII and secret blocking (section 3).
5. A run-level log (section 7).

Then work through the rest by risk tier: a read-only internal agent needs far less than one that touches customers or money.

For how these controls work in Leanroute, see AI governance, agent control and AI cost control. For the practical side of section 5, read how to track AI agent costs per run. For data rules, see LLM guardrails at the gateway.