Part 1 set out the five failure modes that kill AI projects between the demo and production: brittleness, the black box, state amnesia, security theater, and testing impossibility. Diagnosis is the easy half. This is the other one.
Each failure has a known answer. The difficulty is never any single answer — it is that production demands all five at once, and each is a substantial system in its own right. Here is what each one actually takes.
Brittleness → a model broker, not an API call
The fix is not a retry loop. It is putting every model call behind a gateway that knows about more than one provider: model groups that span providers, a selection strategy per group, automatic failover when a provider degrades, and quality-of-service tiers so a cheap background task and a customer-facing request are not competing for the same capacity.
The part teams underestimate is credential and policy lifecycle. If provider keys live in application config, then rotating one, adding a provider, or dropping a deprecated model becomes a coordinated release across every service. Centralising that is what turns a routing layer into an operational capability.
The black box → tracing that reaches business context
Distributed tracing of the whole request tree is the floor: which agent ran, which model it called, which tools fired, what came back, how long each took. Most observability stacks will give you that.
What they will not give you is the attribution that makes a trace actionable — which application initiated it, which workflow stage consumed the tokens, whether the cost came from the original request or from error recovery, and which owner is accountable. A trace that ends at "the model was slow" is a fact. A trace that ends at "the enrichment step in the support assistant retried four times, and it lands on the support budget" is a decision.
State amnesia → a persistent entity graph
Chat history is not state. Production agents need durable, typed business objects with relationships between them — the claim, the customer, the document, the decision and who made it — that survive a restart and can be inspected months later.
Two things make this more than a database. First, working memory attached to entities, so a long-running workflow resumes rather than restarts. Second, purpose-specific projections over the graph, so an agent reads what its task needs instead of everything that exists. Without the second, the context window becomes the bottleneck long before the model does.
Security theater → identity that names real users
Sandboxed execution and secret management are necessary and insufficient. The gap in most agent systems is that the platform has no model of the application's end users — only administrators, developers, and service identities.
Without that model you cannot express the rules that actually matter: this specialist may use the assistant but may not authorize a payment; this agent may read approved policy documents but not drafts; a refund over a threshold needs a manager. Add canonical principals and groups, put access control and human approval on the request path, and route data access through a governed proxy that reflects who is asking and why — without making every end user a database principal.
Testing impossibility → determinism you can switch on
The core move is being able to turn the nondeterminism off. Model mocking and response caching let the same input produce the same output, so a failing test means a broken change rather than a differently-moody model. That one capability is what makes AI code reviewable on the same terms as everything else.
Above it sits evaluation for the cases where you genuinely want to measure behaviour rather than pin it: suites that run against a deployed bundle, with LLM-judged semantic assertions for outputs that are correct in substance but never identical in wording.
The Infrastructure Gap
Between "call the OpenAI API" and "production agent system" is a massive infrastructure gap. It is the AI equivalent of the gap between writing a SQL query and operating a production database -- except the tooling ecosystem for the latter has had decades to mature, while the former is months old.
Most teams end up building this infrastructure from scratch. They spend months constructing a model routing layer, then months more on an observability stack, then months on a state management system. By the time they have something that resembles production-grade infrastructure, they have burned through engineering budget, accumulated technical debt, and their original business requirements have shifted.
The uncomfortable truth is that the infrastructure to run AI agents reliably is harder to build than the agents themselves.
This is not a reflection of poor engineering. It is a reflection of the sheer breadth of capabilities that production AI systems require. No single team should have to solve model routing, state management, observability, security, and testing from first principles every time they want to deploy an agent.
A Platform Approach
The pattern across every successful production AI deployment we have studied is the same: the teams that succeed are the ones that treat infrastructure as a first-class concern from day one, not something to bolt on after the demo works.
This is the philosophy behind FireFoundry. Rather than asking every team to reinvent the same infrastructure wheels, we provide one operating environment spanning four lifecycle stages: Reliable (multi-provider routing, failover, capacity management), Observable (full telemetry, tracing, and replay), and Operational (entity graph, security framework, sandboxed execution).
The goal is straightforward: let engineering teams focus on the agent logic that is unique to their business, while the platform handles the production infrastructure that is common to every deployment.
Building AI agents that actually work in production is hard. But most of that difficulty lies in infrastructure, not intelligence. With the right foundation, the path from demo to production does not have to be a twelve-month odyssey.
Ready to move from demo to production?
Explore how FireFoundry provides the infrastructure layer your AI agents need to run reliably at scale.