How AI workloads affect enterprise network design

Different traffic patterns with AI workloads

AI workloads do not have a single traffic profile. Training, fine-tuning and inference use accelerators differently, which changes how much data moves across the network and how sensitive the workload is to bandwidth, latency and congestion.

A traffic pattern depends on how the model is divided across the compute cluster. In data parallelism, each GPU works on a different portion of the training data and then synchronizes its updates with the others. Tensor, pipeline and expert parallelism split the model or its computation across multiple GPUs instead; these techniques require frequent communication among them.

This makes training traffic highly synchronized. Google has observed sub-millisecond, line-rate bursts in AI training workloads as GPUs finish compute phases and begin exchanging data across the network.

Even if one network path is delayed, the other accelerators involved might have to wait before the next stage of computation can begin. For this reason, training depends on more than raw link speed.

A network might provide enough average bandwidth but still experience queue buildup or congestion when many accelerators transmit simultaneously. Predictable latency and the ability to sustain large east-west transfers become important metrics.

Training also creates substantial traffic as datasets and model states move between storage systems and the compute cluster. Network teams need to plan for accelerator-to-accelerator communication alongside storage and orchestration traffic. Fine-tuning models produces many of the same traffic patterns but at a much wider range of scales.

Inference, where the AI model applies what it's learned to generate results, is more sensitive to how fast applications can send data to a model and receive the answer. For example, real-time applications might depend on consistently low latency, while high-volume inference services need sufficient throughput to handle simultaneous requests. In generative AI, the additional metrics, such as time to first token and tail latency, can also become important.

Inference is also becoming less of a north-south traffic generator as larger models are distributed across multiple GPUs or servers. Retrieval-augmented generation can add further traffic among inference services, vector databases, storage systems and enterprise applications.

The shift from north-south to east-west traffic

North-south traffic enters or leaves the data center as employees access applications, customers connect to services, or systems communicate with the internet and cloud. But in AI data centers, east-west traffic remains within the data center or cloud environment where the data moves among servers, GPUs, storage systems and other infrastructure services.

This traffic pattern places pressure on traditional hierarchical network designs. For example, a three-tier network connects servers through access, aggregation and core layers. These designs can use oversubscription because the bandwidth available to servers exceeds the uplink capacity.

With synchronized AI traffic, however, many servers attempt to use those uplinks together. AI clusters, therefore, adopt flatter leaf-spine (or Clos) fabrics with lower oversubscription. In this design, servers connect to leaf switches, while each leaf connects across a set of spine switches.

This creates multiple paths between endpoints and reduces the number of network tiers traffic must cross. Some large AI fabrics go even further toward non-blocking designs, where the network has sufficient capacity to support a much larger simultaneous allocation of traffic injected by the connected systems.

Google's Virgo Network architecture, for example, uses a flat two-layer, non-blocking topology with high-radix switches to provide scale-out bandwidth for accelerators. This makes bisection bandwidth increasingly important. It describes how much traffic the fabric can carry simultaneously between two network segments.

Therefore, network teams must consider how many accelerators can communicate simultaneously, how much traffic they inject into the fabric and how much contention the workload can tolerate.

Lossless networking and GPU performance dependencies

When congestion causes packets to be dropped, higher-layer protocols such as TCP can retransmit the missing data and adjust the sending rate. For many enterprise applications, an occasional retransmission increases latency.

But this step can generate a substantial consequence in a distributed AI cluster. If congestion delays one participant, the other GPUs involved in that operation may have to wait. To that end, low-loss or lossless networking is a must-have for enterprise AI workloads that are tightly connected across accelerators.

The goal of a lossless network is not to eliminate congestion, but to prevent it from causing packet drops. This requires coordinated buffer management, congestion control and flow control.

Operating such networks requires comprehensive visibility. For example, a 400 GbE link that appears underutilized could still experience short bursts that fill queues and disrupt a training operation.

How enterprises should plan an AI-ready network

The following steps can help organizations build an AI-ready network.

1. Begin with the workload. A large, distributed training cluster, a fine-tuning environment and an enterprise inference service each places different demands on the network.
2. Determine how many accelerators need to communicate, where models and data will stay, if inference is centralized or distributed and how fast those requirements will scale.
3. Understand where that traffic will travel. Map the data paths among compute, storage, applications and external services. This will determine if the organization needs separate networking domains.
4. Capacity planning should focus on where bandwidth is required. Consider the connectivity available to each server, the combined rate at which nodes can inject traffic storage, throughput, uplink capacity, oversubscription and redundancy.
5. Headroom is important because AI traffic can be bursty. The appropriate oversubscription level for the link utilization depends on how much contention a particular workload can tolerate.
6. Decide how much operational complexity you are prepared to support. High-performance Ethernet fabrics using Remote Direct Memory Access over Converged Ethernet requires detailed congestion management, telemetry and configuration. The choice depends on the available networking skills, tools and operational processes.
7. Observability plays a strong role, monitoring everything from queue behavior, packet drops and explicit congestion notification and priority flow control events to latency and congestion hotspots. This helps network teams correlate network behavior with GPU utilization.

Comprehensive planning treats network spending in relation to the value of the compute it supports. The right networking for AI workloads architecture is the one that provides enough bandwidth, latency consistency and resilience for the workload without affecting performance.

Abhishek Jadhav is a technology journalist covering AI infrastructure, semiconductors and advanced computing systems.