Part of:The impact of underused data center infrastructure
AI inference is changing data center design requirements
AI inference requires distinct infrastructure compared to training, focusing on low latency, appropriate hardware based on workload, and tailored power and cooling methods.
Much of the current expansion of AI infrastructure has focused on large GPU clusters designed to train foundation models. These deployments concentrate substantial computing resources and require high-speed interconnects, electrical capacity and advanced cooling infrastructure.
But as AI applications move into production, inference introduces different infrastructure requirements. Inference uses trained models to process requests for enterprise assistants, customer service, document processing and automated workflows. These workloads can vary in computing hardware, response-time requirements, geographic location, demand patterns and rack density.
Infrastructure designed for large training clusters will not always be the most economical or appropriate option for every inference deployment. Data center operators and infrastructure leaders need to reassess how they design facilities and plan capacity to accommodate these different workload requirements.
How inference workloads influence hardware selection
Large-scale AI training relies on continuous computational throughput and high-speed communication among AI accelerators. However, inference has different infrastructure requirements because it must process incoming requests with low latency.
Smaller models and some enterprise applications can run on CPUs. These processors support preprocessing, request handling and application orchestration. By contrast, larger generative models require GPUs for their parallel processing capabilities and high memory bandwidth.
Factors such as model size, memory requirements and the number of simultaneous requests determine how much computing capacity an application requires. For example, a classification system that processes infrequent requests does not require the same infrastructure as a large language model (LLM). However, some high-volume inferences can still require substantial accelerator capacity.
For infrastructure teams, the starting point is the application's expected traffic, response time targets and processing requirements. These factors help determine if conventional servers, accelerator-equipped systems or larger GPU clusters are the right choice.
Preparing power and cooling for AI inference
Rack power density is the electrical load concentrated in a cabinet. More operators are reporting peak rack densities of 30 kW or higher, according to Uptime Institute's 2026 survey.
Nvidia's GB200 NVL72 rack consumes approximately 120 kW at full load, which shows why operators cannot use average facility rack densities to size every inference deployment. They need to analyze the electrical distribution at the rack, row and facility levels.
Inference demand varies with application traffic and concurrent requests. Capacity planning should therefore account for peak loads, electrical headroom and potential future expansion. Operators should avoid relying solely on average utilization as a metric.
Cooling requirements are another crucial consideration, and the right choice depends on the amount and source of heat generated. Air cooling is suitable for most CPU-based, lower-density accelerator configurations, whereas higher-density GPU installations require direct-to-chip liquid cooling.
Higher-density deployments can increase computing capacity in limited space.
Schneider Electric emphasizes the importance of coordinating rack, power and cooling designs. This requires reassessing existing cooling capacity, heat distribution and the feasibility of targeted upgrades. Facility-wide cooling capacity may also mask thermal limitations at individual racks.
Higher-density deployments can increase computing capacity in limited space. However, electrical and cooling modifications must be tailored to the specific workload requirements.
Plan inference capacity around latency and demand
Installed megawatts, available racks and server utilization indicate the extent of the infrastructure. But they do not reflect how effectively it can serve AI requests.
For several enterprise AI applications, time to first token measures how long users wait before a response begins. Throughput indicates how many requests or output tokens the infrastructure processes over time.
Data center operators must balance these metrics against peak concurrency and maintain sufficient spare capacity to accommodate demand spikes. The cost and energy consumed per completed request also affect overall capacity efficiency.
Infrastructure leaders should plan capacity based on workload performance, demand, geographic distribution and utilization.
Location is another key consideration. Placing inference systems closer to users and enterprise applications can reduce network delays, and centralized facilities enable pooling of expensive hardware across workloads to improve economies of scale. Regional and edge deployments address strict latency and data locality requirements.
However, distributing capacity increases infrastructure duplication, networking demands and the complexity of maintaining consistent monitoring and operations. Infrastructure leaders should plan capacity based on workload performance, demand, geographic distribution and utilization.
Where existing data centers fit into AI inference
Existing colocation facilities that cannot support dense GPU training clusters may still accommodate lower-density inference systems for AI infrastructure. The suitability of these facilities depends on the intended workload.
The Akamai Anthropic agreement reflects this broader demand. Anthropic committed $11.6 billion over seven years for CPU-based cloud capacity, with a potential $9 billion expansion. The company plans to work with multiple colocation providers.
For colocation operators, evaluating such an opportunity begins with assessing available electrical capacity and the power available to individual racks. Additional considerations include supported rack densities, cooling capabilities, network connectivity and proximity to customers.
Some facilities may also require targeted electrical or cooling upgrades, which operators must evaluate against expected customer demand, revenue potential and disruption to existing tenants.
Bruce Bateman, chief analyst for semiconductors at Omdia, a division of Informa TechTarget, sees inference as a potential upgrade path for traditional data centers that cannot compete with large training facilities. "If a traditional data center did upgrade, I would upgrade for inference," Bateman said.
He suggested that operators could dedicate a portion of their facilities to inference workloads serving existing enterprise and e-commerce customers. Operators should know that these upgrade investment decisions should not be based on assumptions about future AI infrastructure.
AI inference requires a diverse range of infrastructure considerations to support enterprise applications. Data center design should align with each workload's compute density, electrical and cooling requirements, response-time targets and geographic considerations.
Before committing new capital, data center leaders should assess the capabilities of existing facilities and prioritize upgrades according to actual demand.
Abhishek Jadhav is a technology journalist covering AI infrastructure, semiconductors and advanced computing systems.
