Getty Images/iStockphoto
With Nvidia GPUs, CoreWeave hits architectural milestone
TechTarget has learned that CoreWeave has put powerful multi-rack capabilities into production around Nvidia's latest AI platform.
CoreWeave has brought seven Nvidia Vera Rubin NVL72 racks into production across two regions, totaling 504 GPUs, as the AI cloud provider begins scaling Nvidia's newest systems beyond a single rack.
The company will unveil Wednesday that it has brought up multi-rack Vera Rubin systems on CoreWeave Cloud, alongside new storage capabilities designed to keep data moving as AI workloads scale.
For data center operators and IT buyers, multi-rack Rubin changes the infrastructure planning problem. A deployment must account for network capacity, power delivery, liquid cooling, storage performance and failure handling across the same workload. Those pieces can determine how much of the advertised GPU capacity a customer can actually use.
CoreWeave's Corey Sanders, senior vice president of product, said the hardest parts of moving from a single NVL72 rack to multiple racks were networking and operations.
The networking must provide enough throughput to keep AI workloads from waiting on data moving between racks. The operational challenge grows as workloads span racks that need to function together.
"The operational complexity grows more than multiplicatively," Sanders said in an interview.
Each NVL72 rack combines 72 Rubin GPUs with 36 Vera CPUs, Nvidia NVLink 6, ConnectX-9 SuperNICs, BlueField-4 DPUs, high-speed storage and liquid cooling.
"The single rack is Nvidia's product. The multi-rack domain is the operator's product," said Stephen Sopko, practice lead for semiconductors and deep tech at HyperFrame Research.
Making the network part of the system
Inside an NVL72, Nvidia NVLink provides the high bandwidth scale-up fabric. Connecting multiple racks introduces a scale-out network to handle large amounts of synchronized traffic.
CoreWeave says its architecture uses two-tier, non-blocking connectivity with multiple rails and planes. Each Rubin GPU has two ConnectX-9 SuperNICs, providing up to 1.6 Tbps of scale-out network connectivity per GPU.
"The network is no longer simply connecting servers, it is becoming part of the computer itself," said Sameh Boujelbene, vice president and analyst at Dell'Oro Group.
Once GPUs span multiple racks, Boujelbene said, the scale-out network needs to keep traffic predictable under massive, synchronized workloads. Congestion, load balancing, latency and fault isolation become more important.
She called the 1.6 Tbps per GPU figure a meaningful architectural milestone because it reflects the bandwidth requirements of next-generation AI systems.
CoreWeave says its architecture can scale to roughly 128,000 GPUs per rail. That figure is an architectural-scale target, rather than a claim that CoreWeave has deployed a 128,000 GPU Rubin cluster. The company has disclosed 504 Rubin GPUs across its seven production racks.
CoreWeave also deliberately disables faster GPU-to-GPU links during testing and forces traffic through the backend network.
Sanders said the exercise lets CoreWeave see how traffic moves across the multi-plane fabric and exposes problems involving switches, cabling, software and other network components.
The company validates systems from individual GPUs and nodes through complete racks and multi-rack workloads. It runs GPU diagnostics, interconnect checks, compute and thermal workloads, rack-level 72 GPU jobs and distributed workloads before systems enter production.
Sanders said failures can originate in GPUs, networking, switches, cabling or software. CoreWeave can hold systems from production when failures or additional validation requirements emerge.
Large-scale training is particularly sensitive to the network, Sanders said. Mixture of experts inference also creates substantial inter-rack communication as models become more complex and workloads use multiple components or agents.
Managing the rack as a system
Networking is only part of the infrastructure CoreWeave must coordinate. Power, cooling and hardware health also need to track workloads across racks.
CoreWeave has extended its Mission Control product with Racky, its unified rack manager, and Valvey, its programmable per-rack liquid cooling valve assembly. The Rack LifeCycle Controller handles rack-level lifecycle operations.
Sanders said that CoreWeave manages power and cooling through a combination of hardware and software. Racky and Valvey can monitor infrastructure conditions, adjust cooling flow and manage usage patterns at the rack level.
"The single rack is the operational unit," Sanders said.
The approach allows CoreWeave to manage multiple racks as a larger operating environment while keeping the physical boundaries out of the customer workload.
Sopko said the operational layer is critical as AI systems become more distributed.
"The cabinet is an API object, not a custom build," he said.
A GPU can slow down. A network connection can become unreliable. A switch can fail. A cooling problem can throttle an entire rack. Any of those events can affect a distributed workload even when most of its GPUs are operating normally.
"The job has to finish," Sopko said. A slow GPU, hot rack or congested network path can stall a long running workload and increase the cost of producing each token.
Keeping data close to the GPUs
CoreWeave is also adding storage capabilities aimed at reducing data movement as distributed AI workloads scale.
Its CoreWeave AI Object Storage service includes the Local Object Transport Accelerator, or LOTA, which provides a compute local NVMe endpoint for cached object data.
CoreWeave claims LOTA can reduce latency by eight times compared with reading directly from traditional object storage. Sanders said the comparison is between direct reads from an object storage system and access through the local cache.
The company also says LOTA can provide up to 7 GBps per GPU. The throughput can be sustained under the right workload and cache conditions, Sanders said.
The technology is aimed primarily at large scale training, where model weights, datasets and checkpoints can generate heavy storage traffic. It can also support some inference workloads and frequently accessed data.
CoreWeave is adding cross region write acceleration for workloads such as checkpointing. Data is written durably in the local region and acknowledged locally, while the object data moves to a second region in the background.
CoreWeave says the migration typically completes within 72 hours.
Sanders said the model works for many checkpoint scenarios because the remote copy does not have to be immediately available. The workload can continue while CoreWeave handles the second copy in the background.
CoreWeave is also introducing an Archive tier for data that can move out of the active working set. Archived objects can be read without a separate restore operation, although reads have higher latency and lower throughput than data in the standard tiers.
