Until about two years ago, enterprise AI mostly ran in the background. Models took weeks to train. Internal documents got indexed overnight. Analysis jobs ran on a schedule rather than in response to a live user. If something was slow or briefly unavailable, almost nobody noticed, because almost nobody was waiting on the result in real time.
That has changed. By 2026, AI sits in the path of the customer. Voice agents pick up the phone and handle support calls. AI-powered chat is built into the customer-service flow at major airlines and banks. Loan underwriting decisions are increasingly made by a model in the seconds between a borrower hitting submit and the page reloading. The model itself is still the headline capability. But the actual product, from a customer’s point of view, is now a live service that has to work, every time, the moment someone interacts with it.
That shift has changed the reliability bar for AI. The always-on dependability that enterprises expect from systems like online banking or payment networks now applies to AI as well. But the systems underneath enterprise AI were not designed to that standard, and the discipline of operating AI in production is catching up. The gap is showing up in three places: where users interact with the AI, in the data the AI depends on, and in the compute infrastructure underneath.
The Application Layer Is Unforgiving
Real-time voice is the canary. Most software absorbs latency. A dashboard that takes 800 milliseconds to load instead of 500 is fine. A spreadsheet that recalculates a second slower than expected gets a sigh and continues to function. Voice does not work that way.
“A conversation is a tight feedback loop,” says Anoop Dawar, Chief Strategy Officer at Deepgram. “Add 300ms of delay and the caller pauses, unsure if they were heard. Add 500ms and they repeat themselves. Now the system is processing stale input while the caller talks over it. The conversation doesn’t just slow down. It unravels.”
That changes what reliability has to mean at the application layer. Most software is judged on how it performs on average, or on how it performs across the slowest one in twenty interactions (a measure engineers call p95, or the 95th percentile of response times). For a normal web app, those numbers are usually fine. The slowest one in twenty users sees a slow page load, sighs, and moves on. For voice, that same one-in-twenty interaction is a broken conversation. The bar voice AI has to clear is closer to the slowest one in a hundred (p99), or for the highest-stakes use cases, the slowest one in a thousand (p99.9). A voice turn has roughly 500 to 800 milliseconds of total time before the human notices something is off, and that budget has to cover speech recognition, the database lookup, the model’s response, the synthesized voice, and two network hops. One slow component on one turn is enough to derail the conversation.
Distributed-systems researchers have understood this for more than a decade. Jeffrey Dean and Luiz Barroso made the canonical version of the argument in “The Tail at Scale” in 2013, when they showed that rare slow responses become the dominant user experience once a request fans out across enough services. The original case study was Google search. The new case study is a person on a phone call.
“In voice there’s no spinner,” Dawar says. “There’s just silence. Infrastructure problems don’t surface as error codes or slow page loads. They surface as awkward human moments.”
Voice is the most demanding case. The principle generalizes. Any AI workload that sits inside a synchronous user interaction inherits the same constraint. The model can be perfect. If the inference path takes a second longer than the user expects, the experience is broken.
The Data Layer Wasn’t Built For This
Underneath the model is the data, and underneath the data is a different set of problems. Modern AI systems pull live information from internal databases on every interaction. A chatbot answering a customer-service question is, behind the scenes, looking up that customer’s recent orders. A retrieval-augmented model is reading from a knowledge base of internal documents. Models that update themselves continuously are reading and writing to data stores in real time. All of this assumes the underlying databases are responsive, consistent, and always reachable.
Increasingly, those databases run inside Kubernetes, the open-source software that has become the standard way to run modern applications across servers and clouds. Kubernetes is excellent at managing stateless workloads, the kind that can be killed and restarted without losing anything important. It was not originally designed to manage databases, which by definition store information that has to persist, and even with later additions to support stateful workloads, it does not natively understand the way a live database has to behave.
“Kubernetes can restart a pod,” says Don Boxley, CEO and co-founder of DH2i. “It doesn’t understand the implications of doing that for a live database. Scaling, failover, and replication have to be handled with awareness of the data itself, not just the container it’s running in. If you treat data like stateless compute, you will break things.”
The deeper issue is that monitoring at the wrong altitude misses the failures that matter. Most teams are still watching at the node or instance level. Problems originate at the database layer minutes before they surface upstream. In a batch ML pipeline, those minutes are tolerable. In a customer-facing AI service, they are the difference between a degraded interaction and a lost customer.
The five-year-old definition of high availability was binary. The system was up or it was down. The new definition is granular. The system is preserving the conditions that allow real-time AI to perform on every interaction, or it is failing to. Boxley’s argument is that the orchestration layer does not understand state, the state layer is where the failures now live, and most enterprise teams are looking in the wrong place.
The Hyperscaler Default Is Cracking
One layer below sits the infrastructure question, and here the economics have changed faster than most enterprises have noticed. The default deployment path for any AI workload over the last decade has been to swipe a credit card at one of the three large hyperscalers and spin up GPUs. That path was reasonable when AI was experimental and bursty. It is becoming expensive and rigid as AI becomes steady-state and customer-facing.
“Companies don’t really choose hyperscalers,” says Richard Copeland, CEO of Leaseweb USA. “They default to them. There’s a difference. Costs start to drift. Architectures get shaped around what the platform offers, not what the business actually needs. Over time, you’re not really designing infrastructure anymore. You’re adapting to someone else’s menu.”
The economics that bite hardest are GPU pricing for steady-state inference and data egress fees. Inference is the AI workload growing fastest. It runs all the time, on predictable load patterns, against the same data. That is the workload profile that has traditionally favored dedicated infrastructure rather than on-demand cloud. The silent killer, as Copeland puts it, is egress: when the data needed for retrieval lives in one cloud and the inference runs in another, every retrieval is a billable transit.
The frontier labs have already absorbed this lesson. The $100 billion AWS-Anthropic structure, in which Amazon committed five gigawatts of dedicated Trainium capacity to run Claude, is the largest public acknowledgment that frontier inference at scale wants dedicated infrastructure rather than on-demand capacity. The specialist neocloud build-out, which has produced multi-billion-dollar campuses from Core Scientific, Applied Digital, Lumen, and Nebius over the past year, is the same lesson absorbed at the next tier down. Hybrid is becoming the default architecture for AI: hyperscaler for burst and orchestration, dedicated infrastructure for the steady-state workloads where the economics live.
“The cloud market doesn’t shrink,” Copeland says. “It fragments. Hyperscalers remain dominant, but no longer exclusive.”
What It Adds Up To
Each of these layers is absorbing a different part of the same shift. AI has moved into production, and production has standards. Real-time voice, stateful data, and predictable inference economics are familiar problems. Telecom carriers, transaction processors, and core banking systems have been running production-grade reliability against versions of these constraints for decades. The AI stack is now subject to the same standards. Most of the operational discipline that exists for traditional production systems has not yet been built for AI.
The companies that close the gap first will look less like ML teams running experiments and more like infrastructure teams running core systems. The reliability metrics will be tail-latency budgets and continuity at the data layer. The architectural commitments will be hybrid by default. The operating model will assume that AI is not an experiment in a research environment but a customer-facing service that has to stay online.
The capability has shipped. The discipline has not. That is the work of the next couple of years.
–
By the Control Plane Editorial Team