Having the right tool can make any job easier. That’s a truism. What that old saying doesn’t account for though is that some tools are more specific than others. Meaning, no substitute will do when you need the specific tool, but the frequency that you need it is rare. It’s like having a Dremel tool in your garage, a seam gauge in your sewing supplies, or an immersion blender in your kitchen: you can’t just “swap in” a whisk when you need an immersion blender, but let’s face it, you’re not using it to make a PB&J.
I bring this up because I spent a large portion of my career as a security consultant: specifically, in situations that involve penetration testing. More often than you might expect, I found myself advocating to clients (to the dismay of salespersons in the room I’m sure) that they are probably better off not conducting a pentest but doing something else instead.
This will probably be a contentious point for some, but there are situations where a penetration test isn’t the best use of an organization’s resources. For example, consider an environment that has recently been breached but hasn’t yet done any remediation work. Assuming limited budget, what has most value: a pentest or something else like better detection or hardening the environment? In this case, pentesting would be like using your Dremel tool to pound in a nail. Could you do it? Maybe… but is it the best use of your time and energy?
Of course, it should go without saying that I’m not suggesting there aren’t situations where a pentest is absolutely the best tool for the job. Quite the contrary. Not only can pentesting be specifically required for regulatory compliance reasons (e.g., PCI DSS requirements 11.3.1 and 11.3.2), but it can also deliver tremendous value when used in the right situation and with discipline and forethought. Key to having that be true though means being an educated consumer about pentesting service offerings. Because understanding when and why to use a pentest means you’re using resources most effectively and getting the most value from the test.
What is (and isn’t) a pentest?
As a starting point, it’s useful to point out that service providers aren’t fungible. Meaning, any two firms selling pentesting services might have vastly different service offerings. For example, one provider might sell a simple vulnerability scan under the moniker of a “pentest.” Others might include validating physical access restrictions at facilities, while others might include “war walking” or wardialing instead. Getting the value you want and expect from the test, therefore, involves two elements: 1) understanding specifically what you intend and expect from the test and 2) knowing specifically what the service provider is offering (and getting them to articulate it to you).
Understanding the first element is probably the easiest of the two. What makes a penetration test different from another service (e.g. an architecture review, a vulnerability scan, etc.) is that it’s designed to simulate attack conditions against your environment – either a subset of the environment or the whole thing. Meaning, the engagement team will try their level best to break into your stuff.
Note, though, that this isn’t the same as hiring someone to provide you with a list of the things wrong with your environment. Consider evaluating an application: if you just want to know what issues the application has, you might use a static or dynamic application testing tool, you might put the application through a threat modeling exercise, or you might do some sort of structured testing like input fuzzing. An application-focused pentest, on the other hand, will let you see if there’s a path in but it won’t evaluate all possible paths. In short, that pentesting answers the question “can the adversary get in,” not “what are all the holes I need to patch.”
This means that having a clear goal and expectations in mind at the outset is critical to getting maximum value. Having this understanding will tip you off when a pentest isn’t the optimal route – for example when you already know your environment has weaknesses or where you suspect it is likely to. On the other hand, don’t underestimate the impact of the “wow factor” associated with someone demonstrating “hands-on” how your organization can be compromised. There are goals where that outcome can be huge – for example if you’re looking to “sell” a request for budget up to the executive leadership chain. Likewise, if you have an environment that you have invested heavily in from a security point of view and you want a “stress test” to see how it performs against attackers, a pentest might be a good way to accomplish that.
What are the goals?
The second thing that’s important to understand is what the service provider will be delivering as part of their testing process. There is quite a bit of variability here in the marketplace, so it’s important to be educated here. It’s also well within bounds to ask whether a service provider follows a specific methodology such as the penetration testing execution standard (PTES), NIST SP800-115 (“Technical Guide to Information Security Testing and Assessment), or (depending on what you want to test) a more specialized process such as OWASP’s Application Testing Guide.
Any reputable service provider should be comfortable explaining to you – in a high level of detail – specifically what they will do, the methods they’ll employ, the tools they’ll use (again, it’s in your rights to ask for a list), and details about the scope of the engagement. If they’re not willing to open the kimono – or worse yet give you a blank stare if you ask them about the standards enumerated above – this speaks (in my opinion) directly to their credibility.
Another technique – which again a reputable provider will agree to – is to have a representative of the organization observe the testing process. This is a good idea anyway as it can help build understanding and knowledge of attack techniques among the staff that participate, but it’s also advantageous to have someone there who can respond to any issues that arise.
Either way though, having a clear understanding of specifically what the service provider will provide in conducting their test is helpful. It’s helpful so that you can evaluate service providers in an “apples to apples” fashion, it’s helpful in combination with your self-evaluation and assessment of your test goals, and it’s helpful so that you know you’re getting a reliable, reproducible, and objective test at the end of the process.