Why most agent demos collapse in production

Every agent demo I have seen in the last two years follows the same arc. The room is impressed. The prototype books a meeting, writes a summary, or reconciles an invoice. Someone says “this would save us thousands of hours.” A pilot is approved. And then, somewhere between month three and month nine, the initiative quietly disappears from the steering-committee agenda.

The post-mortems always blame the model. The model was almost never the problem.

Demos optimize for the happy path. Production is the unhappy path.

A demo is a system observed under the most favorable possible conditions: clean inputs, a friendly operator, a scenario chosen because it works. Production is the opposite — adversarial inputs, ambiguous requests, and a user who does not know what the system cannot do.

The demo question is “can the agent do the task?” The production question is entirely different: what happens when it doesn’t? Who notices, how fast, and what does it cost before they do?

Most teams cannot answer that question, because answering it requires infrastructure the demo never needed: evaluation suites, action logs, permission boundaries, rollback paths. That infrastructure is 80% of the work and 0% of the demo.

The three collapses

In my experience production agent initiatives collapse in one of three ways, in descending order of visibility.

The trust collapse. The agent makes a visible mistake in front of the wrong audience — a hallucinated number in an executive report, an email sent to a customer that should never have been sent. One incident of this kind erases months of accumulated credibility. The system may be right 97% of the time; the organization only remembers the 3%. This is not irrational. A junior employee who was wrong 3% of the time, confidently and in writing, would also lose the room.

The economics collapse. The pilot ran on twenty users; production means two thousand. Suddenly someone in finance is asking why the inference bill has a comma in it, and nobody modeled cost per task, cached anything, or routed easy requests to cheaper models. Agentic systems have a cost curve, and a system that is worth deploying at demo scale can be strictly not worth deploying at production scale.

The ownership collapse. The quietest one. The prototype was built by an innovation team, and when it worked, no one could answer the boring questions: who is on call for it? Who reviews its permissions? Who updates it when the API underneath changes? A system with no owner degrades until someone turns it off — usually citing one of the first two collapses, which by then have both happened.

What the surviving systems share

The agentic systems I have shipped that are still running share four properties, none of which involve model choice.

They have deterministic boundaries: the set of actions an agent can take is enumerated, permissioned, and enforced outside the model. The model decides among allowed actions; it does not get to define the set.

They are evaluated before and after every change — a regression suite of real tasks with known-good outcomes, run the way software teams run tests, because that is what it is.

They expose their reasoning to correction: when a human overrides the agent, the override is captured and feeds back into evaluation. The correction path is a feature, not an embarrassment.

And they have an owner with a budget — a person whose job description includes this system, and a cost line someone reviews monthly.

The uncomfortable summary

None of this is exciting, which is exactly the point. The demo is the easy part now; the models are good enough that impressive is cheap. What separates the initiatives that compound from the ones that collapse is the operating system around the agent — guardrails, evaluation, economics, ownership.

Probabilistic systems can be built in a weekend. Governable ones take a year. If your plan does not contain that year, you do not have an agent strategy. You have a demo.

Related: Deterministic Guardrails, Agent Evaluation Frameworks, Production Patterns for Agentic Systems