New application security risks lead IT teams to DevSecOps

SEATTLE -- Tools and technical tricks abound to improve product security, but as organizations embrace DevSecOps, the human factor remains the most important, and the most challenging.

DevSecOps, once the domain of early adopters at tech companies, has reached mainstream IT shops thanks to a flood of high-profile security breaches that made headlines in 2018 and 2019, from consumer-facing firms such as Starwood-Marriott, Panera, Newegg and Capital One to tech companies such as Elasticsearch and Lenovo. Such breaches have only increased in frequency and impact -- the July "2019 MidYear Data Breach QuickView Report" by security research firm Risk Based Security tallied more than 3,800 security breaches in the first six months of the year alone, affecting 4.1 billion records -- the worst year on record.

"Every week there are breaches in the news, and it's brought security toward the front of people's minds," said Olly Ewert, a developer of security architecture at a New Zealand-based SaaS company. "There's still a long way to go before every developer is thinking about security every step of the way, but you don't necessarily need that -- all you need is one or two looking at every pull request with security in mind."

Automated tools that build product security tests into the CI/CD process and tools that templatize security as code for developers are important to implement DevSecOps, but they only go so far.

Everyone, starting from the front door of the building, needs to be educated about things like social engineering and compliance rules.

"Right now, everybody is just rushing -- they know they need security, but there are new tools every year and new problems to solve every day," said Rohit Sharma, associate director of technology, security and privacy at Protiviti, a management consulting firm based in Menlo Park, Calif. "Anyone can use a tool, but they need to understand its purpose and the architecture of the organization."

Tools alone can lead to confusion rather than clarity when they deliver conflicting results about vulnerabilities, and tools can't prioritize how product security risks are addressed.

Thus, enterprises have begun to put into practice the idea that security is the job of everyone within an organization, whether technical or nontechnical, Sharma said.

"It's becoming a more critical part of everything -- everyone, starting from the front door of the building, needs to be educated about things like social engineering and compliance rules," Sharma said.

A seat at the table -- DevSecOps collaboration expands

Once organizations gain a sense of urgency about improving IT security, the next problem is where to start. At large organizations, security and risk assessment specialists help to set strategy, but at some smaller organizations, DevOps pros and site reliability engineers (SREs) step in.

"Ops and SREs are where our [DevSecOps] transformation is happening," said Michael Silva, a DevOps engineer based in Seattle.

Silva's role over the last nine months at a digital media organization with just over 100 employees has been to act as a conduit between an imperative to improve product security from upper management and the junior developers who must work it into an already overflowing feature backlog. So far, that has taken the form of developer training in basic security best practices and code templates developers can use to secure app deployments.

"We can walk developers through, 'Here's how you go from A to Z with best practices,' and, 'Here's how you can go from A to L and then use the template,'" Silva said.

SREs have also begun to get the proverbial 'seat at the table' during developer sprint planning sessions, where feature requirements and defect lists are drawn up. Silva has "dabbled" in AppSec in the past, he said, but seeing certain patterns through ITOps and DevOps experience is also helpful to set a product security baseline.

"There are things you learn as a sys admin and DevOps engineer that are just bad practice; you know where the threat vectors are, and those patterns repeat themselves," he said.

DevSecOps challenges require the art of persuasion

Challenges remain as companies embrace DevSecOps. The DevOps imperative of release velocity, for example, acts as both a motivator and an inhibitor to improving product security. "Shifting left" and designing security best practices into applications by default ultimately makes life easier for everyone, but finding the time for such an undertaking often feels impossible to developers tasked with ever-increasing demands to deliver revenue-generating features.

Where IT security pros once ruled with an iron fist, halting the software release process in its tracks when risks were discovered, they must learn to act as facilitators. Instead of dictating what product security tools will be deployed, security pros in DevSecOps environments must learn what developers will actually want to use.

This type of organizational change doesn't come easily, but progress is possible, Ewert said.

"When I started four years ago, there was a real 'us vs. them' mentality between developers and security," he said. "It took us three years to get the culture to the point where they approach us [for help]."

That shift came about through training sessions and explanations for why certain security rules and practices are necessary, Ewert said. The SecOps team also found unobtrusive ways to dovetail its efforts with the rest of the organization's; they piggybacked on an IT monitoring project that rolled out standardized monitoring agents across the organization to get security monitoring agents installed, for example. It has also been helpful for Ewert's team to frame security recommendations in terms of how they can make the application and infrastructure more efficient, such as by avoiding costly distributed denial-of-service attacks.

"We only block a release if it's absolutely critical. Whenever possible, we put things into the developer backlog instead of saying, 'Stop everything and fix this right now,'" Ewert said.