When IaaS provider Hivelocity, based in Tampa, Fla., ordered 20 data center switches from Juniper Networks in July 2021, the vendor said Hivelocity could expect the new hardware within a month. But a month became six weeks, which turned into two months, which became five months.
"Our current ETA is now March of 2022," COO Steve Eschweiler said. "It's crazy." Hivelocity's procurement manager recently ordered 110 additional switches, he added, which Juniper said won't arrive for nearly a year.
Eschweiler's team is in good and growing company, with countless organizations seeing escalating lead times on networking products, including data center switches, campus switches, routers, firewalls and wireless access points (APs). Andrew Lerner, analyst at Gartner, said typical lead times of three to six weeks for network switch orders have ballooned to six to nine months or more.
"We are getting calls from our client base saying, 'Hey, what's going on?'" Lerner said.
What's causing the global chip shortage?
The network hardware delivery delays stem from the global chip shortage, which has limited the availability of items ranging from PCs to cars. Experts blame the pandemic, saying the shift to remote work increased reliance on the cloud and spurred both enterprise and consumer tech buying, supercharging the demand for chips.
In mid-2021, as organizations prepared for a return to the office, network infrastructure purchases spiked and put additional pressure on an already stressed market. Meanwhile, on the supply side, COVID-19 outbreaks have halted or slowed production at semiconductor manufacturing plants worldwide.
But other factors may have exacerbated the shortage. Forrester Research Vice President and Research Director Glenn O'Donnell wrote in a recent blog post that tensions between the U.S. and China have prompted Chinese technology companies to hoard chips -- fearing imminent American sanctions -- further draining the global semiconductor supply.
O'Donnell also pointed to industry consolidation trends that predated the pandemic, with only a small number of companies currently operating chip manufacturing facilities worldwide. That's a problem with no quick fix, as building a new semiconductor fab costs billions of dollars and takes upward of two years.
Sameh Boujelbene, senior research director at Dell'Oro Group, added that supply problems tend to provoke more demand, thus worsening shortages and creating a vicious cycle.
"It's human nature: Supply issues drive panic purchasing behavior," she said.
In the case of the chip shortage, network procurement managers are reportedly preordering and stockpiling routers, switches, APs, firewalls and more, ultimately increasing competition for gear and exacerbating market conditions. But Boujelbene said she understands the impulse and would do the same in their shoes. "You have to do it, as a procurement manager," she said. "If you don't, you could lose your job."
Network upgrade plans under threat
Rural telecommunication carriers have already had to delay their planned network upgrades for up to a year as a result of the global chip shortage, according to the Rural Wireless Association (RWA). RWA members started seeing long lead times on network gear in the first half of 2021, the organization said in comments to the Federal Communications Commission.
At Hivelocity, the networking team has a stockpile of gear that will enable them to complete all active projects, Eschweiler said, including a recently inked deal requiring them to build out 25 new data centers in 90 days. But, with business booming, thanks to customers' pandemic-fueled digital transformation initiatives, he said he expects to run out of their on-hand inventory in early 2022.
"It certainly could hinder our ability to expand," Eschweiler said, noting that, for every 40 servers Hivelocity sells, it deploys two new top-of-rack switches. "At a certain point, we won't be able to stand up additional racks, let alone open new locations."
In addition to long lead times, Gartner's Lerner said he's seen price increases as high as 25% to 30%, with most markups hovering around 10% to 15% as of the end of 2021. Vendors are strategically funneling what chips they do have toward equipment with the highest price tags and biggest profit margins, he added.
"If you want to buy a high-end modular switch that has a $25,000 list price, you're much more likely to get that in a reasonable time frame than a very low-end switch that's $1,100 or $1,200," Lerner said.
7 tips for surviving the global chip shortage
Experts generally expect semiconductor supply challenges to last until at least mid-2022, with some predicting the market fallout will stretch into 2023 and beyond. Given that somewhat grim forecast, here are seven strategies for network teams as they navigate the ongoing chip shortage.
1. Plan to wait
Like it or not, organizations must take longer product lead times into account in their network lifecycle management for the foreseeable future. Expect to wait at least six months for any new gear, period, Lerner said. "It's pretty dire in the sense that there's not a whole heck of a lot you can do."
2. Share upgrade plans with vendors
One thing enterprises can do, according to Lerner, is proactively share plans for future network infrastructure upgrades with their suppliers as early as possible. "Work with them," he advised. "If you know that, in a year, you're building a data center and you'll need switches, tell them now."
This step is especially important for large orders. An early heads-up helps vendors more accurately forecast demand and could help them better support customer timelines.
3. Make some noise
Lerner also noted that, in his experience, suppliers are not currently prioritizing customers on a first-come, first-served basis. "With vendors, the squeaky wheels are getting the grease," he said. "Your size of spend, how much noise you make and how competitive the situation is will absolutely impact when you get your stuff."
A loyal Cisco shop, for example, might not get its orders as quickly as a customer that seems likely to jump ship for Juniper, or vice versa. "I am finding that organizations that apply competitive pressure are doing better," Lerner said.
That might mean procurement managers tell Vendor A that, rather than buying a switch with a six-month lead time, they plan to consider competing products from Vendors B, C and D. "Suddenly, Vendor A says, 'Well, maybe we can get it to you in five months,'" Lerner said. "Competition is your friend in helping you move to the front of the line."
4. Switch vendors -- if it makes sense
If enterprises can find an alternate manufacturer with significantly shorter lead times, then changing brands might be worthwhile. The odds of finding someone able to deliver gear relatively quickly may be slim, however, as virtually every network hardware maker is grappling with the global chip shortage. For example, Eschweiler noted that, while Hivelocity is having a hard time getting Juniper switches, other organizations are waiting on Cisco.
Even so, Eschweiler said his team will consider working with a new vendor if push comes to shove. "I'll tell you this: We do not want to slow down our growth," he said. "If we have to make a vendor switch temporarily, we'll do it to be able to put our customers online."
5. Recommission old gear
As organizations wait months for new hardware, some have had to get creative with what they have on hand. At Host Duplex, a managed cloud hosting provider based in Chicago, the networking team has resorted to using spare appliances and manually repairing old, previously defective gear while they wait for new edge routers to arrive.
"It's not a method we prefer," founder Sam Jadali said. "Given the exorbitant prices and long waits, however, we're not left with many options."
6. X86 it or go public cloud
For some appliances, such as basic routers, load balancers and firewalls, Lerner suggested forgoing hardened appliances and deploying software on x86 servers. Of course, that assumes bare-metal hardware is available, he added.
As an alternative to deploying more complex appliances, such as data center switches, organizations might consider outsourcing some needs to a public cloud or colocation provider.
7. Shop the gray market
Another workaround is to shop for secondhand gear on the gray market. Given the global chip shortage, however, even used hardware has reportedly become astronomically expensive and limited in availability. Hivelocity's Eschweiler said he saw prices for 1G switches increase exponentially in 2021. He added that it was impossible to find other switches, such as 10G and 40G models, on the gray market at all.
Buying used networking equipment has both pros and cons, so gray market shoppers should conduct due diligence to avoid getting stuck with defective or counterfeit products. And be aware that vendor warranties and guarantees may not apply once a product has changed hands.
While tactics such as those listed above might help network teams make the best of the chip shortage, experts agreed there's only so much anyone can do until big-picture conditions improve. And Eschweiler, like countless other network leaders, has an uneasy eye on the calendar. If lead times don't normalize by the end of 2022, he said, Hivelocity's strong growth will likely stall due to a lack of gear.
"It stinks," Eschweiler said. "Hopefully, things free up sooner than later."