Tiers of Penetration Testing Maturity

2013-08-29 21:09:08 by chort

Today Dave Aitel (presumably in response to a certain company announcing their "0day pentesting partnership") decided to dredge up an old post from Haroon Meer related to 0days and penetration testing. The basic point by Haroon was, what exactly is this testing? The conversation on Twitter brought up some good points, which prompted me to write a longer analysis of why I think most pentesting is a total waste of time.

The way I see it, there are four types of pentests. They can be imagined roughly as a pyramid, with only the top-tier being really useful. I'll attempt to briefly explain them here.

Tier 1: Just the Compliances.

This is exactly what it sounds like. Acme Co. needs a penetration test as part of their regular compliance certification. They call in Cookiecutter Auditing Inc. to do a "pentest" and walk away with a thick report of NMAP and Nessus output. They promise to "block the portscans on the firewall" and whatever other horse crap makes the auditors check their box, money is exchanged, and the matter is dropped until next year. Value: Checkbox.

Tier 2: Prove We're Secure.

This is similar to previous, only in this case IT Manager Bob wants to prove to senior execs that Acme Co. "is secure." So Bob hires Bargin Basement Llc. to "pentest" a single external IP, that's on a separate firewall interface, with a single server behind it and no ports open. "Just try to hack that server" says Bob. So the Bargin consultant dutifully tries for two days to "hack the server" (which means run NMAP, Nessus, and Sqlmap, then surf reddit for 1 1/2 days), and come back at the end with a report saying something like "we couldn't hack it, but the firewall allows tracert, which is a medium risk," money changes hands, and everyone is happy. Value: Bob looks like a champ.

Tier 3: Prove We're Insecure.

Alice knows Acme Co. has woefully under-spent on security, but management just won't listen to her warnings and certainly won't approve her budget. She hires L33t & Associates to pentest her network for a week. L33t goes to town, throwing down SQLi and leaving USB thumbdrives in the parking lot. After a week they do a "shock and awe" presentation to the exec team, who promptly tell Alice "here's some money, make this go away." Alice buys a NGFW, a WAF, and installs badge readers. Value: Another security vendor IPO gets it's wings.

Up until this point, the tests have provided basically no value. Nothing substantially changes in Acme Co. At best, they buy a few more firewalls that no one takes the time to configure correctly, let alone monitor. The only change is a few people at Acme Co. feel better about themselves. They feel they've "done something" about the problem. There are a lot of flavors of the third tier, by the way. Maybe L33t & Associates comes back the next year and Alice gets to buy DLP and a malware sandbox appliance. Once again it will feel like "the threat was handled, problem found, problem solved." Nothing could be farther from the truth.

Tier 4: Catch Me If You Can.

In this scenario, CSO John calls in Eve & Partners. John explains that they have a critical database of customer information, and a fileshare of secret design documents. He'd like to find out if his team can catch Eve before she and her colleagues can exfiltrate the data.

Eve's team spends a week doing recon on social networks, passive DNS, social engineering emails, etc. Finally they launch their attack. They establish a foothold on the network through phishing messages that trick users into installing backdoors. Simultaneously they login to remote access gateways using socially engineered credentials. John's team notices suspicious non-HTTPS traffic over port 443 and discovers one of the compromised workstations. They analyze the compromised machine, create IOCs for the RAT & other attacker tools, and use those to locate the other affected hosts.

Unfortunately for Acme Co., the remote access gateway use goes unnoticed and Eve's team manages to exfiltrate the customer records from the database. The next day someone notices an anomaly in the netflow data and realizes what's happened.

Eve & Partners give a report highlighting the need for better monitoring and alerting on access gateways and John's team starts writing a log-analysis plugin to implement it. Value: Detection and Responses processes were exercised, gaps were found, and steps to remediate them were initiated.

Do you notice what the difference is in Tier 4? The first three tiers are entirely passive. They amount to "here's our brick wall, try to get past it." The fourth tier is active. The Acme Co. defenders are actively watching the network and seek out the attackers. The adversaries didn't need to use l33t 0days or resort to gimmicks (physical access) that real-world attackers almost certainly won't use (unless you're a diplomatic post, or uranium enrichment facility, in which case your threat-model is different).

I'm not saying penetration testing is always a waste of time. On the contrary, if you have built what you think is solid detection and response capabilities, you absolutely should test them. The point is testing exploits against your static defenses is useless. You'll never have a static defense effective enough that removes the need to use active countermeasures, and if you keep going down the "buy appliances until it's fixed" rathole you won't ever have money for the most effective investment, which is more people.


I should point out that at work, we're happy customers of Tenable, Immunity, and DSquared (I love small vendors). The point isn't to bag on the tools, it's to highlight that results are often mis-applied. I'm also not gazing down from a tower of smugness where we've built all the capabilities I describe, because we aren't there yet. I'm merely pointing out that people and process are far more important than technology.

Add a comment:




max length 1000 chars