AI Agents in the Enterprise: Where They Work, Where They Fail, and How to Decide
Your support automation reads a ticket, queries the customer record, and prepares a refund. The demo works. Then someone asks the question that changes the system: “Why not let it issue the refund too?”
That is the point where an AI feature becomes an operational risk decision.
I have seen architecture discussions jump from model accuracy to agent frameworks while skipping the decision that matters most: what may this system do when its interpretation is wrong? The answer should determine whether you need a chatbot, a deterministic workflow, a copilot, or an agent. The technology label comes later.
My rule is simple: choose the least autonomous operating model that preserves the value of the use case. More autonomy is not more mature. It is more authority delegated to a system whose behavior cannot be specified through code paths alone.
Start with control flow, not the word “agent”
Section titled “Start with control flow, not the word “agent””Most enterprise “agent” proposals combine four different system types. Separating them prevents you from paying the cost of autonomy when ordinary software would do the job better.
This classification comes after the portfolio question, not before it. If the initiative has no measurable outcome or usable process, apply the elimination criteria in Where Does AI Actually Create Value? before debating architecture.
| System type | Who controls the next step? | What it should do |
|---|---|---|
| Chatbot | You | Answer a question or continue a conversation |
| AI-assisted feature | Application code | Classify, extract, summarize, or generate content inside one defined step |
| Deterministic workflow | Predefined rules | Execute a stable sequence with known branches |
| Agent | The model within set boundaries | Select tools and actions to pursue an objective |
Anthropic’s guidance on effective agents makes the same architectural distinction: workflows follow predefined code paths, while agents dynamically direct their own process and tool use. It also recommends starting with the simplest design because agentic systems exchange latency and cost for flexibility.
That distinction changes investment and governance. If refund-policy-v3 already defines eligibility, amount limits, and approval rules, encode those rules. Use a model to interpret an ambiguous customer message if needed, but keep the decision path deterministic. Giving an LLMLarge Language ModelA neural network trained on large volumes of text that generates, summarizes, and transforms natural language. control over known rules does not create intelligence. It creates another failure mode.
An agent earns its place when the path cannot be enumerated in advance and the system must adapt across tools or incomplete information. Repository diagnosis is a good example. The system may need to inspect logs, search code, reproduce a failure, choose a test, and revise its plan. Hard-coding every path would defeat the point.
That flexibility still lands inside your delivery system. The AI Paradox explains why faster code generation can move the bottleneck into review, testing, and recovery.
Flexibility is the benefit. Loss of predictable control flow is the price.
Four operating models, not a maturity ladder
Section titled “Four operating models, not a maturity ladder”I use four operating models to make the delegation explicit. A single product can use more than one. Your support system might act as an assistant for policy questions, a copilot for refunds, and a supervised agent for low-risk ticket routing.
| Model | System authority | Suitable boundary |
|---|---|---|
| Assistant | Proposes; you perform the action | Analysis, options, drafts, queries |
| Copilot | Prepares an action; you approve execution | Refunds, messages, configuration changes |
| Supervised agent | Executes inside a bounded scope | Sandboxed code changes, reversible routing, controlled remediation |
| Autonomous agent | Executes and reports | Low-impact, observable, reversible actions with proven evaluations |
The autonomous category should remain exceptional. It is useful when stopping for approval would destroy the value and when mistakes can be detected and reversed cheaply. Renaming a low-confidence ticket queue may qualify. Changing production access rights does not.
Do not assign one level to an entire application. Assign a ceiling to each action. Reading a customer profile, drafting a reply, issuing a €20 credit, and disabling an account carry different consequences. One agent identity with one broad permission set hides those differences exactly where you need them exposed.
Set the autonomy ceiling where errors become real
Section titled “Set the autonomy ceiling where errors become real”A polished demo tells you that the happy path exists. It does not tell you how much authority to delegate. Score the proposed action against the conditions below.
| Decision factor | Lower-autonomy signal | Higher-autonomy signal |
|---|---|---|
| Objective clarity | Success depends on unstated judgment | Completion and failure are explicit |
| Reversibility | Effects cannot be fully undone | Effects can be rolled back quickly |
| Environmental uncertainty | Tools, policies, or context change without notice | Inputs and tool behavior are constrained |
| Error cost | Failure affects money, access, safety, or rights | Failure has a small, contained impact |
| Tool and data quality | Permissions are broad; data is stale or ambiguous | Tools are narrow; data lineage is known |
| Observability | You only see the final output | Every action, input, and result is traceable |
| Interruptibility | Work cannot be paused safely | Timeouts and a stop path leave a safe state |
| Evaluation | Success is judged by impression | Task and harm metrics run on realistic cases |
One bad dimension can cap the whole action. A production-access agent may have a clear objective and excellent logs, but its error cost and privilege scope still make autonomous execution a poor starting point.
This risk-based approach is not just my preference. The NIST AI Risk Management Framework Core calls for explicit task mapping, documented human-AI roles, defined oversight processes, and controls for external AI components. For high-risk systems, Article 14 of the EU AI Act ties oversight measures to risk, autonomy, and context, including the ability to override, reverse, interrupt, or stop the system.
The practical consequence is direct: “human in the loop” is not a control specification.
Put approval at the action boundary
Section titled “Put approval at the action boundary”Human approval helps only when you can make an informed decision before the effect occurs. A modal that says “Approve?” after hiding the evidence is ceremony.
For a refund copilot, the approval view should show the source ticket, matched policy, customer history used, proposed amount, confidence or evaluation signals, and the exact API action. For a coding agent working on payments-api, review should happen at the pull request. The agent can edit in a sandbox, run pytest, and open fix/KN-20-timeout; branch protection prevents it from merging or deploying.
I prefer controls that reduce authority before they try to detect misuse. Give the agent a dedicated identity. Grant access per tool and per action. Enforce transaction limits outside the model. Keep credentials out of its context. Record tool inputs and results in a trace that an operator can reconstruct.
Then add runtime containment: action budgets, retry limits, timeouts, rollback procedures, and a kill switch that does not depend on the agent cooperating. The OpenAI practical guide to building agents recommends escalation after failure thresholds and before sensitive or irreversible actions. The OWASP agentic AI threat model explains why model-level instructions are not enough: tool misuse and identity or privilege abuse are system risks.
These controls limit the blast radius. They do not prove the agent is reliable.
Promote autonomy from evidence, not confidence
Section titled “Promote autonomy from evidence, not confidence”An agent should earn authority through measured operation. Start in shadow mode or as an assistant. Compare its proposed actions with real outcomes. Move to copilot only after you know which failures reviewers catch and which they miss. Grant supervised execution to a narrow set of reversible actions, then expand one boundary at a time.
Measure both task success and harm. A support agent that closes 92% of tickets but issues incorrect credits to 3% of customers is not “92% reliable.” Those errors have different costs. Track invalid tool calls, policy violations, unnecessary escalations, rollback frequency, time to detection, and reviewer disagreement alongside completion rate.
Tie those signals to cost per completed outcome, including review and recovery. Measuring AI ROI provides the baseline and cost-per-task method for that calculation.
You also need failure tests that attack the control model. What happens when a ticket contains prompt injection? When the customer record is stale? When the refund API times out after processing the transaction but before returning confirmation? When the agent reaches its action budget halfway through a case? If you have not tested those states, you do not know the safe autonomy ceiling.
Anthropic’s study of agent autonomy in practice describes autonomy as a property of the whole deployment, shaped by model behavior, product design, and oversight. That is the right unit of analysis. Changing the model can alter performance, but permissions, checkpoints, tools, evaluation, and rollback determine what a failure can do.
Do not confuse monitoring with certainty either. In the SHADE-Arena evaluation of sabotage and monitoring, the strongest evaluated sabotage agent completed a hidden harmful objective without detection in 27% of tasks under the study’s monitor setup. That is not a production incident rate. It is evidence that a detailed trace and another model watching it do not make harmful behavior impossible.
Make the decision on one real action
Section titled “Make the decision on one real action”Pick one proposed agent use case and write down the first action that changes a real system. Not “handle support.” Use “call POST /refunds for up to €50” or “open a pull request against payments-api.”
Score that action across the eight decision factors. Choose the lowest operating model that still delivers the outcome. Name the identity, permission boundary, approval point, trace, budget, timeout, rollback path, kill switch, and evaluation that support it.
If you cannot name those controls, keep the system in assistant mode. If you cannot measure whether the action was correct, do not automate it. Run the decision again only after production evidence changes the answer.