Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Is 25 Gigabit Ethernet the cure for data center oversubscription?

With reduced hardware costs and more flexible lanes in the physical architecture, 25 Gigabit Ethernet may offer answers in oversubscribed data centers.

"The more things change, the more they stay the same." That's what everyone has always said, that's what conventional wisdom has always held, and that's the mantra those of us who have been in the networking industry long enough have held as our religion. Not anymore.

Today, our new talisman against oversubscription may become 25 Gigabit Ethernet (GbE).

Networking speeds and feeds, for instance, always incremented in fairly predictable ways on the LAN: 10 Mbps gave way to 100 Mbps, which gave way to 1 Gbps and so on through 10, 40 and now 100 Mbps. While not exciting, and not always welcome for those of us who had to buy optics to support these increases, it was all fairly predictable.

In that same vein, wireless was increasing its speeds over time, but it always lagged behind wired clients at the access layer. Most wired clients these days are connected at 1 Gbps, as that seems to be the sweet spot for now, and nobody wants the expense of upgrading cable plants, NICs, etc., just so the funny cat videos can arrive faster on end users' screens.

We now find ourselves at something of a crossroads. For the first time since its advent, consumer wireless has outpaced the generalized ability of our infrastructure to support it. Most wireless radios are connected back to an access port, via a Cat-5e cable running at 1 Gbps. Some cable plants have different cables, perhaps even Cat-6. All common cabling, however, has a 1 Gbps maximum throughput.

With wireless radios now moving into 802.11ac running at more than 1 Gbps, a problem starts to emerge. Because today's radios offer higher and higher densities, the backhaul connection from each radio to the wired switch fabrics are severely oversubscribed. Can 20 different people connect to a radio at 1 Gbps each and reasonably expect any kind of performance from a radio with only a single 1 Gbps connection itself? Nope.

Retooling existing cables to provide needed throughput

Enter two competing standards proposals: for 2.5 Gbps and 5 Gbps speeds over those same Cat-5e and above cables. Both standards have an impressive list of industry stalwarts behind them, and both are aiming to become IEEE standards. Multiple vendors are rumored to have pre-standard versions in the works, and you can bet that every wireless vendor who doesn't yet will as soon as possible.

Hardware costs are significantly reduced while, at the same time, overall speeds are increased at a critical bottleneck: the server to top of rack connection.

Existing cable plants are expensive to rip and replace, with some estimating the cost to average around $300 per cable. With tens of thousands of cables in any given large enterprise, this equates to a significant cost to move to more capable cabling like Cat-6a or fiber. Add to that the fact that wireless radio manufacturers aren't champing at the bit to provide 10 Gbps support on their radios, and you can quickly see why new standards for faster speeds on existing cables would be so welcome.

Backhauling radios at as high a speed as 5 Gbps would allow for much more reasonable data center oversubscription rates, considering the current densities of the average wireless radio. All of this coming at a price point much cheaper than a cable plant replacement project. Of course, since these are not even IEEE standards yet, and no word has come down from any manufacturers suggesting how they'll approach the problem, we can't really speculate on pricing. It is likely, however, that some amount of capital outlay will be required to make use of these new speeds. Yet, even if we had to rip and replace all wireless radios and all backhauled access switches, the price will still likely be cheaper than doing the same for cabling. Wireless, however, is not the only area where we are seeing these shifts and changes.

Need for 25 Gigabit Ethernet could transform the market

As data centers have necessarily become more efficient, and density (measured by number of virtual machines, processors, or generalized compute and storage capacity) has increased, one thing has become clear: We have too many cables, optics, and competing standards and speeds to make for efficient and portable design. This is where an emerging standard, 25 Gigabit Ethernet, can make its mark. Because of its more efficient use of lanes at the physical layer, hardware costs are significantly reduced while, at the same time, overall speeds are increased at a critical bottleneck: the server to top of rack connection.

A very common data center design is to have a switch sitting at the top of each rack, aggregating all Ethernet traffic in that rack. The so-called top of rack (ToR) switch is then connected to an end of row aggregation switch, and all of those are then backhauled to the core. CLOS design -- or leaf/spine -- changes this a bit, but the concept is fundamentally the same: The access layer for servers gradually come together into increasingly capable devices to move traffic in and out of the data center. With servers stuck at 1 GbE or 10 GbE today, and ToR switches at 40 GbE, the hardware expense climbs quite quickly due to the need to meet a certain satisfactory level of oversubscription. Proponents maintain 25 Gigabit Ethernet aims to fix that problem by connecting the servers to the ToR at much faster speeds, and allowing the ToR to backhaul at 50 Gbps or 100 Gbps (or additional multiples thereof.)

Industry heavyweights support 25 Gigabit Ethernet

The 25 Gigabit Ethernet evolution is in its infancy. But backers -- which include most of the industry's heavyweights from Cisco, Juniper, Broadcom and HP to Marvell and ZTE -- are bullish on its goals and future, in part because they claim the standards effort would reduce ToR density by a nearly 4:1 ratio in many cases. Given the number of backers, the seriousness of the design effort, and the fact that the IEEE voted midway through last year to start a working group, this is definitely not an effort to ignore. While not something we'll likely see on servers anytime soon -- at least, not as an official standard -- my money is on this succeeding and becoming the de facto standard for servers in the next few years.

Which gets me to the beginning of this piece: Things are definitely changing. And as much as that might be disquieting to those who have sat back and coasted on the predictability of innovation, the reality is that all of these emerging proposals and standards are good for the industry. Will all of them pass from idea to wide adoption? It's hard to say, but the fact that we are trying to solve networking problems more quickly now than ever before is a testament to the lessons we've learned from the past. Stagnation, the status quo, more of the same, these are all ideas we must banish if we want to move the industry forward. Embrace the changes.

Next Steps

The 2.5, 5 GbE roiling market

Wireless growth fuels wired upgrade

Getting ready for gigabit speeds

This was last published in October 2015

Dig Deeper on Network Infrastructure