WEKA addresses AI inference bottlenecks with WEKApod 3, its first custom-designed hardware, paired with NeuralMesh 6 software to improve GPU efficiency.
Weka has designed its first custom appliance from the ground up, advancing its full-stack infrastructure strategy that aims to maximize GPU use and minimize storage bottlenecks.
This week, the AI data and memory company unveiled the WEKApod 3 appliance and NeuralMesh 6 microservices architecture software.
WEKApod 3 marks a departure from the storage vendor's previous reliance on third-party OEM chassis. Combined with the new NeuralMesh software platform, Weka promises up to 10 times higher token throughput for AI inference workloads -- addressing critical challenges as AI deployments scale.
"[Training] gave the industry a relatively stable problem: large, predictable data flows, long-running jobs -- we know how to keep the GPUs fed in that instance," said Jim Sherhart, vice president of product marketing at WEKA. "Inference is just a completely different problem, and it changes everything that you would think about from a performance perspective and from an infrastructure perspective. The workloads are much less predictable."
That unpredictability is driving WEKA's shift toward vertical integrations. The company aims to optimize its entire stack for AI inference workloads.
"It's interesting that WEKA is now designing its own hardware system," said Simon Robinson, an analyst at Omdia, a division of Informa TechTarget. "In the past, it was more focused on the software. And WEKA would leave it to its partners to put the hardware together."
This change highlights the degree to which the market is moving, Robinson said. Instead of assembling a system themselves, customers can buy into a custom-made system that's optimally configured.
The shift might also partly be due to current supply chain constraints. Customers need high-performance AI storage systems, but also need to ensure that the performance and capacity customers do have is used efficiently -- which is easier to achieve using purpose-built systems.
According to Robinson, training is typically a throughput problem. Massive amounts of data are needed for numerous steps in the process, such as for model checkpoints -- a snapshot of the model at a particular moment in time. If the system crashes, the model can be rolled back from a checkpoint. Checkpointing in particular requires a lot of storage and throughput, but is relatively predictable.
Inference, however, is much more about I/O, as the process is much less predictable. Unlike training, which involves large, steady dataflows and long-running jobs, inference requests are highly variable. An end user might have a quick interaction with ChatGPT that uses minimal resources, or they might need multiple agents to run simultaneously -- taking up hundreds if not thousands of tokens.
"You don't know what the request sides are going to be," Sherhart told TechTarget. "Context needs to survive across those requests."
GPUs have limited high-bandwidth memory and finite DRAM. When those fill up during periods of high demand, the system must forcibly remove an active inference task from memory to make room for incoming requests. And when this happens, the GPU must recompute the context for that task again.
An idle GPU is a sin, as I've heard.
Simon RobinsonAnalyst, Omdia
"While the GPUs look busy, they actually end up doing a lot of re-computations that don't actually result in more outcomes," Sherhart said. "Storage performance is no longer the right measure or metric anymore. The question really is, 'How efficiently can I feed the GPUs and keep those long context windows going?'"
Inference is also about latency.
"An idle GPU is a sin, as I've heard," Robinson said. "You're paying an awful lot for this resource. If you're just waiting for storage, that's one tax. If you're making a calculation you've already made previously, that's another tax."
Two years ago, according to Robinson, it was all about GPU performance. Now, the bottleneck organizations are facing is quick and efficient access to data in the storage layer.
How WEKApod 3 and NeuralMesh 6 address GPU efficiency
WEKA's latest releases directly address the GPU efficiency challenges that AI inference workloads face. Instead of relying on third-party chassis, WEKA designed hardware optimized for keeping GPUs fed with data.
An image of Wekapod hardware module that slots into a chassis.
WEKApod 3 is designed for density. It comes in three configurations -- Nitro, Prime and Prime Max. Nitro is focused on performance; Prime is a middle-ground offering focused on performance and capacity; and Prime Max is focused on capacity.
Prime Max can fit up to 70 NVMe drives and has 1.1 exabytes of capacity. Prime Max also features 10.2 TB/s throughput and 210 million IOPS."
Sherhart also adds that by controlling component sourcing directly and not relying on third-party vendors, WEKA gains a practical advantage in today's constrained supply chain environment. While industry lead times for storage components can stretch out four to six months, WEKA's integrated approach enables the company to deliver hardware systems rapidly -- a factor that Sherhart says influences customer decision making.
"A lot of the deals we've been winning recently have been because we have hardware available and we're able to deliver within weeks," he said.
While WEKApod 3 provides the physical foundation, NeuralMesh 6 software introduces capabilities designed to address inference workloads. NeuralMesh 6 is WEKA's sixth-generation software that operates as a distributed mesh of services across storage nodes. The distributed design is useful for AI inference. By distributing data across different nodes, NeuralMesh 6 enables parallelism while reducing latency, which GPUs need to stay productive, according to the vendor.
An image of the NeuralMesh dashboard featuring statistics such as physical and usable capacity, throughput, IOPS, alerts and performance.
AI clouds and GPU-as-a-service providers will typically serve multiple clients simultaneously, making isolation and, therefore, multi-tenancy an important feature.
"Having really strong multi-tenancy and security here is really important," Robinson said.
NeuralMesh 6 enables multi-tenancy through bare-metal resources or virtual isolation options.
"A lot of times, storage providers will do it one way or another, so customers have to compromise," Sherhart said. "We decided to do it both ways."
Although other vendors like Vast Data exist in this space, WEKA was one of the first vendors to address GPU and storage bottlenecks with KV caching, an approach Nvidia has since validated, according to Robinson.
"This is the moment for WEKA to make the most of it," Robinson said.
Robinson noted that while token throughput and efficiency are interesting, what matters is how it translates to revenue.
"The CIO is hugely aware of ROI," Robinson said "There's been lots of headlines and studies around underwhelming ROI. They're making these big bets, and the returns can be opaque, so they need to know that whatever they're deploying is being deployed as efficiently as possible."
Alexander Gillis is a Technical Writer and Editor at Informa TechTarget, with more than 8 years of experience writing about technology.
