xy - Fotolia

Upgrading the core when network downtime isn't an option

Network downtime is often a necessary evil for network upgrades, but in the world of 24/7 broadcast media, maintenance windows are hard to come by.

Nobody likes downtime, but it's usually a necessary evil for network upgrades. For most network engineers, however, there's sometimes a small window at some odd hour of the night when few (if any) users would be affected. But in the world of 24/7 broadcast media, those windows are hard to come by. In this edition of The Subnet, we speak with Jess Probasco, senior network engineer at American Public Media, who shares what it's like to run the network for the second largest public radio producer in the United States.

What have you been working on lately?

Jess Probasco: We currently run all of our audio for our broadcasting and recording in the studios with IP multicast, and we just replaced a large core and all of the access switches for that environment. We finished wrapping that up here last month, and now we're trying to look at finishing up the rest of our infrastructure. We've just upgraded to newer [Cisco Catalyst] 3850 switches and Cisco Nexus 7000 core switches.

What drove that investment?

Jess ProbascoJess Probasco

Probasco: Our old environment -- [Catalyst] core 6500s -- was eight years old, and it was at end of life. All of our edge switches were roughly around the same age as well, and we needed to find an environment that gave us high availability and redundancy and also allowed us to take advantage of newer technologies that utilize that environment as well. Now we're going to add virtual servers and [Cisco's] UCS chassis into that broadcast environment -- for the broadcast equipment and servers. We implemented the Nexus 7000 the year prior for our data core, and it just made sense to keep the technology the same between the two environments.

What's it like running a network for a broadcast media company?

Probasco: Downtime really isn't an option a lot of times. As much as we in networking would like to have maintenance windows and schedule downtime, when it comes to the audio side and being on the air, it's really hard to find those broadcast outages you can take. For instance, when we did this new upgrade, we were able to complete it with roughly a five-second outage, swapping out the entire core and the edge switches and everything. Downtime is minimal, and that also drove us to the newer Nexus environment, where there are dual switches, dual cores and high availability.

On a day-to-day basis, there's [the ongoing challenge of] constantly jumping from ISP to ISP with weather outages. Part of the problem is a lot of [our radio] transmitter sites are out in the middle of nowhere, so that one's a little challenging -- trying to get reliable network connectivity to some of these sites, especially after a storm comes through.

The data side seems to be the normal, typical networking [issues]. But the audio side is all multicast-driven, so that gets a little hard to troubleshoot when there are issues sometimes; it's not necessarily a single point that might be failing.

What technologies are you are trying to learn more about these days?

Probasco: Software-defined networking -- it's more programming than traditional switching. In combination with some of the automated tools that you could use to do some of those switch rollouts, provisioning and changes, [there's benefit in] learning some of those scripting languages [for non-SDN platforms]. I know Cisco is big on Python scripting, and [you can use it in] their Cisco Prime and Cisco ACS [products] that we use here. So trying to learn some programming languages is what I'm interested in right now. It's definitely a [trend] that's not going anywhere -- programming in general with networking.

Can you give an example of how you might use programming in networking today?

Probasco: We currently are in the middle of rolling out Cisco Prime infrastructure, so we'll be able to, on a large network, programmably take care of new switch rollouts or be able to offload some of the day-to-day work to maybe help desk or a lower tier. [We want] to be able to say, ‘Hey, someone in cube A or cube B needs access to this VLAN or needs this port changed to a different VLAN.' To be able to give them the ability to submit a ticket through this or do it without having to jump in the switch manually is where we're going with the programmability.

What did you want to be when you grew up, and how did you become a network engineer?

Probasco: Originally I wanted to be a lot of things, like any kid. I started off wanting to be a paramedic and then I changed that to an electrical engineer. I had an opportunity to intern at an electrical engineering shop when I was 15, and I decided afterwards that wasn't quite the path I wanted to go. I didn't want to run electrical cables all day and work with the circuitry. So I started looking at computers and playing around with them, and one thing led to another. I got into voice originally, and from voice I came into networking when VoIP came around. That's when I had to learn networking, and I really fell in love with networking and what you could do with it.

If you were a superhero, what would your special power be?

Probasco: That's a tough one. I guess if I was a superhero, I would like the ability to be in more than one place at once.

That sounds like it would come in handy at work.

Probasco: Yeah, but not even just at work -- it would be in general, in life. Sometimes you overcommit yourself, and it would be nice to be able to have powers to be in multiple places.

This was last published in August 2014

Dig Deeper on Network management and monitoring