Planning AI infrastructure upgrades: Four steps for success

The importance of AI infrastructure upgrades

Planning and funding AI infrastructure upgrades help businesses address several key present and future challenges.

Prevents and overcomes bottlenecks

Modern machine learning (ML) models and AI platforms require massive computing power, vast storage resources and high-performance networking technologies, such as server-to-server InfiniBand. This is critical for intensive model training and real-time inference, such as queries and analytics. Traditional IT architectures often can't meet these demands, resulting in computational or data transfer delays that can impair AI performance and productivity. Timely infrastructure upgrades ensure AI systems get adequate resources and optimized performance.

Promotes scalability

Data sets and model complexity are growing daily. Traditional IT infrastructures often lack the scalability needed to easily expand and integrate new resources and services. Well-considered infrastructure upgrades facilitate a modern, flexible environment that scales on demand and ensures seamless integrations without downtime or other operational disruptions.

Manages power and cooling

Power and cooling significantly impact the operational costs of AI. These platforms demand massive computing power, typically using advanced, power-hungry GPUs and other processors. These systems require costly power and intensive cooling to protect hot servers. Infrastructure upgrades provide energy-efficient hardware and advanced cooling technologies, such as immersion cooling and direct liquid-to-chip techniques, that can lower costs and enable higher thermal densities across the infrastructure.

Enhances security and governance

Security and compliance issues increasingly affect AI. Large data sets contain proprietary content that must be secure and comply with regulatory requirements regarding provenance and application. AI platform behavior is increasingly scrutinized for appropriate use, the accuracy and effect of its outcomes, and the transparency and explainability of how outcomes are achieved. It takes careful consideration and comprehensive governance to meet regulatory requirements and answer legal entanglements. Infrastructure upgrades can help deliver the systems and services needed to bolster data security and AI governance.

4 steps to prioritize AI infrastructure upgrades

Traditional IT infrastructure upgrades, such as adding servers or storage, are typically reactive processes. But reactive upgrade strategies don't work well for AI platforms.

AI training cycles, inference workloads and token volumes are skyrocketing. According to McKinsey & Company's 2026 report, "The future of AI workloads," inference workloads could account for as much as 40% of data center demand by 2030, which is a CAGR of about 35%.

For AI, this means that reacting to bottlenecks as they appear or adding capacity when needed is just putting a Band-Aid on a bullet wound. AI infrastructure upgrades demand a proactive approach where regular modernization cycles continuously improve computing density, energy efficiency, security and governance. This requires business leaders to align and budget AI infrastructure investments with clear operational goals.

The following four steps are a good start in this direction.

1. Set business priorities and goals

AI platform health, availability and performance affect AI's value. A platform with performance bottlenecks, availability problems, frequent and prolonged training downtime, and other issues degrades the UX. This, in turn, reduces AI adoption and use, limiting the revenue and insight from the investment. Consequently, infrastructure upgrades must be integrated into the AI platform lifecycle and operating budget to ensure AI infrastructure investments align with intended business outcomes.

The following are priorities and goals to assess:

• Understand the demands of AI workloads. Upgrade strategies and priorities will be largely driven by the predominant AI workloads being used. For example, training requires large amounts of computing power and high-performance networking interconnects, while inference benefits from low-latency, highly scalable, edge-centric infrastructures to address user queries.
• Set clear business goals. Business leaders might adopt specific, measurable, achievable, relevant and time-bound goals for AI infrastructure upgrades. For example, business goals focused on cost efficiency would move lesser-used ML models to cloud infrastructure. This would reduce idle compute resources and ensure that the most-used ML models can use the available on-premises computing resources. A focus on scalability might increase the number of API requests while maintaining a maximum latency.
• Select meaningful metrics. Business leaders can use a variety of metrics or KPIs to track the success of AI infrastructure upgrades. This might involve a mix of technical metrics, such as system uptime and model latency, and business metrics, such as processing costs per inference or transaction.

2. Consider existing AI infrastructure

Business and technology leaders must assess the state of AI infrastructure to determine its needs and viability against a set of technical criteria. IT leaders can take the following steps to spot broader compliance, standards and workflows issues:

• Audit the infrastructure and assets. Any infrastructure upgrade strategy must include a comprehensive audit of hardware, hardware maintenance agreements, applications and software licenses. The audit should include a review of system configurations, security implementations and change management practices. Evaluate and identify areas of technical debt, such as a delayed or outstanding network upgrade, and remediate these as part of the AI infrastructure upgrade cycle.
• Review data quality and composition. AI depends on vast amounts of accurate, complete, relevant and timely data. Data resources are always changing; they require continuous examination and enforcement of data quality and format standards. Curate data regularly to remediate aged and irrelevant data and enforce applicable data retention requirements to ensure compliance.
• Optimize storage resources. Vast resources are required to provide the storage capacity and performance capabilities for AI data. Typically, infrastructures organize storage into performance tiers, ensuring the most important and frequently accessed data resides on the highest-performing storage, and less essential data is relegated to less expensive and lower-performing storage. AI training and inference data should be stored on appropriate tiers with adequate capacity, performance and headroom for optimal AI performance. This might require periodically migrating or reorganizing data among tiers.
• Review and expand automation and orchestration. AI depends on high levels of automation and orchestration. For example, automation plays a key role in processing AI data and organizing data pipelines. Orchestration is vital for managing AI workflows. Every AI infrastructure upgrade cycle should include an evaluation of the scope and effectiveness of the AI platform's automation and orchestration capabilities. Automate where possible and optimize orchestration as needed.

3. Optimize hardware infrastructure

With clear goals in mind, business and technology leaders can make effective upgrade decisions for the servers, networks, storage and other parts of AI infrastructure. Upgrades should align with business goals while meeting budget constraints in ways that deliver the greatest ROI.

• Invest in specialized servers. Conventional CPUs are inadequate for efficient ML training and AI inference workflows. AI infrastructure upgrades must focus on replacing aging CPU-centric servers with advanced, power-efficient servers. Newer servers provide multiple GPUs with ample memory and high-performance interconnects for high bandwidth interGPU communication. Servers can also provide high-bandwidth networking interfaces at 400 Gb/s and higher, using techniques such as remote direct memory access and CPU offloading.
• Update network bottlenecks. Upgrade evaluations must look at LAN requirements. Network upgrades should include an end-to-end link capable of supporting high bandwidth, so review all network components in the AI platform to ensure they support the required speeds.
• Update storage bottlenecks. Data needs increasingly fast exchanges with server memory for training and inference. This strains storage capacity and performance. Add capacity as needed to accommodate ample headroom for large data sets. Also, add appropriate storage tiers to balance performance requirements with capacity and cost. Lastly, older storage interconnects must be upgraded to achieve higher throughput.
• Upgrade power and cooling technologies. While energy-efficient hardware lowers power use, the volume of hardware needed for effective AI systems can cause power costs to skyrocket. Consider cost-efficient alternatives. These can include on-site renewable power, such as solar arrays, and regional energy collaboratives that power from local wind or solar farms. Cooling costs can also be extreme, so select servers with cooling efficiency in mind, such as direct-to-chip liquid cooling and liquid immersion cooling.
• Consider composable architecture. One longer-term upgrade path is to replace traditional packed systems with composable architecture. This approach provides a modular environment for GPU, memory, storage, network and other modules. These can be reconfigured, replaced, upgraded and scaled without complete hardware renovations. For example, rather than adding complete servers, a composable infrastructure would add more GPU modules. Composable architecture is based on hyperconverged architectures.
• Rebalance public cloud use. AI platforms aren't an all-or-nothing proposition. Various components exist in different locations. For example, an AI platform might use a third-party large language model from a major provider, while smaller supporting models and autonomous agents are deployed on-premises to manage cost and performance. Weigh the cost versus performance tradeoffs in running different AI components in the cloud compared to running them locally in terms of token costs and API calls.

4. Optimize data security and AI governance

Vast data stores and increasingly complex infrastructures can create serious vulnerabilities that might be exploited. AI hardware infrastructure upgrades must include a review of data security and AI governance postures.

• Establish uniform data security. Data security can shift or become inconsistent as storage resources expand and tiering demands change over time. This makes data security reviews particularly important for AI infrastructure upgrade projects. Implement technologies and practices needed to establish uniform data security, such as data encryption, strong role-based access control, and clear data retention and disposal standards. These practices must support current regulatory compliance obligations and adhere to data sovereignty requirements.
• Establish uniform AI governance. The business must implement comprehensive oversight of the AI system to ensure transparency, observability, explainability, bias mitigation, accuracy, performance and appropriate use. Such oversight is critical for proper ongoing AI operations and addressing the local, national and regional legislation surrounding AI and its use. Deploy one or more AI observability and management tools to support AI governance, as well as a rigorous suite of reviews and validations. AI infrastructure upgrades should include regular reviews and updates of governance tools and practices.

Stephen J. Bigelow, senior technology editor at TechTarget, has more than 30 years of technical writing experience in the PC and technology industry.