ICS security fails the Black Hat test

Industrial control systems hit the mainstream at Black Hat this year, with over two dozen program sessions tackling different angles of the subject. The takeaway: Vendors still aren't really trying.

The news at Black Hat 2018 wasn't great when it came to industrial control systems. But while numerous sessions added up to sweeping condemnation of ICS security, there was at least the occasional saving grace that some vendors will correct some problems -- at least some of the time. Still, the apparent lack of a security-conscious culture within these organizations means they'll only fix the minimum, leaving similar products with the same underlying hardware, firmware and fatal bugs untouched and unsecured.

Speaking in a session, called "Breaking the IIoT: Hacking Industrial Control Gateways," Thomas Roth, security researcher and founder of Leveldown Security, an embedded and ICS security consulting and research company based in Esslingen, Germany, walked through the security faults of a series of five gateway devices he'd found at prices he could afford on eBay. He wanted to look at commonly deployed, relatively current devices -- things you find in the real world.

"If you go out on the network and start scanning, you'll find thousands of these devices. In fact, you'll find entire network ranges that are used almost exclusively for these devices," he said.

"Often, they use static IP addresses with no VPN protection." One device he looked at had a proprietary protocol for its wireless communications. But if you could break it -- and he did -- you had access to every one of those devices in the field, because the network addressing architecture was flat and unsegmented.

The first device he looked at was typical of his various experiments, tackling a Moxa W2150A which connects ICS devices to wireless networks via an Ethernet port on the device side and a wireless interface on the other side. In between the two interfaces is an easily opened case that reveals a circuit board with pads for connecting to a debugging port. Roth discovered, in a common theme across many of the devices discussed at the conference, the port was a serial terminal connection that booted directly to a root shell in Linux.

"This is a design decision, not a bug," Roth said. But he noted that if you have the device and you can access a root shell, then as you are writing exploits, you can debug them directly on the device, "which is a pretty nice situation to be in."

Roth noted the firmware for the device was available on the internet from the Moxa website, but it was encrypted. At first, this seemed like a dead end. But in looking at earlier firmware versions, he noticed one of the upgrades included adding the feature of encrypting the firmware.

This led him to an unencrypted update version, which included a package called "upgrade_firmware." This, in turn, led to a function called "firmware_decrypt" -- a function name that gave the audience a chuckle -- which gave him plaintext access to the current version of the software. The decryption key was, needless to say, included in the upgrade code.

Roth raised an issue that hasn't been much discussed in ICS security: supply chain security issues caused by the wide prevalence of openly accessible terminal access ports on devices. You can change the firmware, he said, write the changed version back to the device, return it to your distributor without mentioning the change, "and they will happily resell it to someone else." In fact, he knows this because he conducted an experiment and was sold a device with firmware he had previously rewritten.

Roth discussed four more devices in some detail, with two of them still in the process of disclosure, "and there are a lot of fun issues."

Beyond Roth's pathway strewn with pwned gateways, there were other such sessions, including ones that found significant vulnerabilities in medical devices, cellular gateways, smart city infrastructure and satellite communications.

Jonathan Butts, CEO of security consultancy QED Secure Solutions, located in Coppell, Texas, noted in a press conference at the event that dealing with vendors around ICS security disclosure had been particularly frustrating. In the case of a pacemaker made by Medtronic, a protracted process leading to the company deciding that changes in the product weren't necessary led Butts and co-speaker Billy Rios, founder of WhiteScope LLC, a cybersecurity company based in Half Moon Bay, Calif., to demonstrate their attack live and let the audience judge for themselves.

"To be honest," Butts said, "after about the one-and-a-half-year mark, and you see stuff like [Medtronic's response], you get fed up."

ICS security: Protection? Not

While it's theoretically possible to protect at least the devices that aren't implanted in human bodies by placing the ICS equivalents of a firewall at strategic network junction points, a session by Airbus security evaluators Julien Lenoir and Benoit Camredon showed a widely deployed ICS firewall made by Belden could be remotely exploited.

The Tofino Xenon device is typically situated between the IP-based control network and local ICS assets that use Modbus, EtherNet/IP or OPC protocols. Interestingly, the device itself doesn't have an IP address; it is essentially invisible to ordinary interrogation on the network.

A custom protocol allows a Windows machine running a configurator to discover and then send configuration data to a Xenon device. The configurator knows the addresses of protected ICS devices and knows the Xenon is somewhere between the configurator and the devices. The Xenon knows to watch for packets that carry a specific payload and recognizes them as packets from a configurator.

The two researchers were able to reverse-engineer the protocol enough to understand the arrangement that was used for encryption keys. The configurator discovers devices using a common key and then generates two additional keys that are unique to the particular pairing of that configurator and that specific firewall. All of these keys could be extracted from the discovery session, and then the keys unique to the device were used to establish a connection with the device.

"We were able to get a root shell," Lenoir told the audience, heralding the familiar theme that almost all ICS devices are actually outdated Linux kernels. "Once everything was running as root, now the appliance was no longer a black box, but was instead a Linux kernel."

From here, they settled on an attack model that used the devices' ability to be updated from files on a USB stick. Camredon explained the updates comprised two files, both encrypted. "One is an update script, and one is a data file that is an image, including an image of the kernel."

It turned out that all configurators and all Tofino Xenon devices used the same key for decrypting the update files. Because they had access to root on the Xenon, they were able to extract this key, at which point they further discovered there were no checks in the update script to ensure the data file hadn't been tampered with since it was created.

Thus, a breached Xenon could be modified in whatever way the attackers wanted, an image of that system made, and the image could be encrypted and included in an update package without the separate installation script detecting the change.

The Xenon has been updated to correct these problems since the researchers disclosed their findings. So, in theory, the firewall is back in business. One problem Roth noted, though, is these systems often come in dozens of variants, with different names and model numbers.

"If you report a bug to some of these vendors," Roth said, "the vulnerability gets fixed, but then there are 10 different devices which run the same firmware, and they are left completely unpatched."

Roth suggested this was a clear indication of the lack of security culture at many ICS vendors.

"It's like exploiting in the '90s," he concluded. "We have no integrity protections on any of these devices."

At another moment, he made a sweeping generalization: "Everything runs as root; everything runs on outdated Linux kernels; everything runs on outdated web servers. If any of these components fails, you have root permission."

Dig Deeper on Threats and vulnerabilities