Manage Learn to apply best practices and optimize your operations.

How to make penetration test results matter

Many organizations that conduct penetration tests do so only to fulfill compliance requirements and satiate auditors' demands. Those same organizations also tend to limit the budgets allotted to pen testers, according to Dave Shackleford, founder and principal consultant of Voodoo Security, which leads to penetration tests that fall well short of mimicking real-world attack scenarios. As a result, enterprises may not be getting the full value for their pen-testing dollars.

"They want to just kind of get it done, and that leaves a lot open, unfortunately," said Shackleford.

In this interview, Shackleford  explained how organizations can get the most out of penetration test results. First, pen tests do need to recreate the experience of real attacks to the greatest extent possible. According to Shackleford, even the most budget-strapped organizations can incorporate relatively inexpensive social engineering techniques to make pen tests more realistic. Shackleford also detailed how to put the results of a pen test into action, namely by prioritizing and uncovering vulnerabilities in a way that makes sense for each individual organization.

Perhaps most notably, Shackelford explained how important it is for an organization to set goals for its penetration test before it's even conducted.

"A lot of organizations don't have a real goal in a pen test. Probably the most important goal for many is to check the box for compliance and make the auditors go away," said Shackleford. "And I don't fault people for that [because] they have to do it, but if you really want to get value, why not say the goal of a penetration test is to get to this sensitive data or these sensitive systems? I just don't see a lot of people doing that at all."

Below is a transcript of the video.

Brandan Blevins: Hi, there, I'm Brandan Blevins with Thanks for watching this video. Joining me today is Dave Shackleford. Dave is founder and principal consultant with Voodoo Security. Dave, it's a pleasure to have you with us today.

Dave Shackleford: Thanks for having me.

Blevins: Dave, most organizations conduct what they may consider to be a penetration test, but would you say they fall short on some key areas that make up an effective pen test?

Shackleford: I think a lot of organizations do, and in some cases I don't think they really know that they do. I think it's really kind of a prescribed thing we've come to realize in the industry. Money factors in, right? People don't want to spend a lot of money, they just kind of want to get it done, and that leaves a lot open, unfortunately, yeah.

Blevins: How closely should a penetration test mimic a real-life attack scenario, and conversely what are the consequences of making a pen test as real as possible?

Shackleford: A great question. And really if you had infinite time and money, you would have the most accurate and realistic penetration test possible. So you would take these advanced threat actors with in-memory exploitation scenarios, and really sophisticated social engineering exploitation, and all of these things and really include that. But people, again, are bounded by time.

They're bounded by money, and so they have to do the best they can, but even then I think they should start adding more of that because if this is intended to be a point in time representation of risk and the real attack surface that's open and possible, if you don't try to emulate what real attackers are doing, then you're not really getting the full picture. So I think that needs to happen a lot more.

Just an example is things like malware. A lot of it attacks incorporate some form of malware that gets dropped, yet nobody includes malware emulation as part of their pen test. That's a huge disconnect because it's not real, right? Most attackers are going to drop something. They're going to do something once they get in. So I think there's a lot to be done.

Blevins: Dave, for those budget-limited organizations out there that want to conduct a thorough penetration test, are there any easy victories you can recommend?

Shackleford: Another good question. I have a lot of those that I work with. Certainly a first thing is to use some sort of social engineering attempts. Again, a lot of people are comfortable with this just because they really feel like they're stepping over some boundary of trust with their employees, but that's how the attackers are getting in. And it's actually not that expensive to do some social engineering, so that's one thing.

I'd argue the second thing that is they should really push for is to incorporate whatever the methodology or tools used in the test are and make sure that whoever is doing a pen test for them puts that into the report so that they can repeat those steps. And then they get longer value from the test because they can go back and they can repeat it themselves and really learn more as an organization as they go along. So those are two very quick things that I think can help.

Blevins: Now once an organization has conducted a penetration test, can you recommend any metrics that gauge the effectiveness of that test?

Shackleford: Yeah, so metrics is a very big discussion, right? And it's kind of a holy war in the world of security as to what the best ones are. Things, for example, depending on the type of test you're doing, the length of time before you're detected. So, for example, if I'm doing a pen test and I manage to compromise the system, how long was I there before you were able to find me, you know, some sort of alarms went off? That's a really good metric that not a lot of people really tend to use.

For example, looking at things like patch management and configuration management as part of the pen test, especially if you're doing it internally where you can measure over some time period, whether it's weekly, monthly, quarterly, yearly even as to gradual reduction in high risk or critical risk vulnerabilities that are there due to missing patches and things like that. So that's a lot of it as well.

Blevins: Once the penetration test has been completed and the effectiveness has been measured, what are the next steps, Dave? How do you go about putting that penetration test into action in terms of security?

Shackleford: So one of the things that you should really make sure you're focused on is the criticality, the prioritization because it all comes down to remediation. This is what we often call closing the loop in that vulnerability lifecycle. A pen test is really nothing more than one type of test as part of that. And so if you go break things or find potential flaws yet never really close that loop and remediate the flaws that you found, then obviously there's a problem there.

And so what tends to be effective, especially for big organizations is to have some kind of ranking system where you say, "Look, these 10 things have to be fixed within this period of time. Everything else, we can kind of put in another bucket and come back to."

And so for any test, I don't care whether it's really mature or less mature, there has to be some results set that gets a ranking, gets some sort of prioritization to it. And you've got to define as a business or an organization how quickly you're going to be able to handle that, and then look to improve it over time. It's gradual improvement of things like remediation time that really define how mature and how well you're doing.

Blevins: Dave, even that prioritization is hard, right? We've got things like CVSS that can help you try to rank vulnerabilities, but what do you advise in that space?

Shackleford: Well, CVSS or any of those types, you know, kind of calculation systems as to criticality or real risk, the danger with any of those is because it could potentially give you either a false sense of security, or the opposite a real false sense of how vulnerable you are. When, in fact, you're not because they have no way to incorporate things like other controls that are helping remediate things, right?

So you can't incorporate three layers of firewalls into your CVSS score. So it's just going to come back and say that's really vulnerable, but what if it's already inside your organization? You have no way to really incorporate that so I think there's a danger in those, but ultimately prioritization is hard. But what you have to look at is basically the attack factor. If somebody came from outside, for example, and they were able to do some real damage coming in, I would always put that higher up on the list versus somebody that's inside already.

At the same time it depends on the type of test. Are you more worried about insiders, for example? Insiders are your really critical kind of focus area for a pen test, then you would turn that on its head. So I can't give you a completely objective answer because it totally varies, depending on the test. And it depends on really what people are most worried about, but you can at least shake out something like, it's a really bad risk. It comes out from CVSS, for example. Now, let's look at how they got to it, and then kind of go from there.

Blevins: So perhaps for that organization that's looking to conduct a penetration test or outsource that service to a third-party vendor, maybe the first step they should take is trying to determine what they actually want out of the penetration test? Is that correct?

Shackleford: Yeah, I would say so. I mean, a lot of organizations don't have a real goal in a pen test. Probably the most important goal for many is check the box for compliance and make the auditors go away, and I don't fault people for that. They have to do it. But if you're really looking to get value then say, well, let's say the real goal of a pen test is to get to this sensitive data or these sensitive systems. And then kind of gauge how well that was accomplished in terms of the output or the outcome of the test.

So coming back to something like prioritization, you could say well, let's say that if the consultant that was doing the test, for example, got this close to the test in the time allotted or got this close to those critical systems in the time allotted, that's a much more successful test even though they didn't necessarily compromise 14 or 15 or 20 other systems along the way. It's all about what you need to get out of it to kind of gauge things.

And so, again, it's a very subjective industry all the way around. I just don't see a lot of people really doing that at all.

Blevins: Dave Shackleford, thanks for joining us today.

Shackleford: Hey, thanks for having me.

View All Videos