If you’ve been in the realm of penetration (“pen”) testing in any capacity for any length of time, you’ve probably experienced the conversations around inconsistent pen testing results across teams or vendors. This isn’t anything new in the pen testing world. The conversations probably ranged from friendly internal team banter to more serious discussions with external vendors on pen testing program success metrics.
Is this a case of mistaken identity? In this article, we examine potential factors and circumstances that could lead to such conflicts and confusion, as well as ways to address the issue.
Vulnerability Assessments vs. Penetration Tests: What is the Difference?
One long-standing tiff that purist-practitioners have had with the larger market forces out there has been a matter of verbiage. The parties cannot agree on definitive definitions for the various components of a vulnerability management program. Obviously, this could be one of the many factors that contribute to the larger issue of pen test results. The most-used terms in enterprise assessment programs are “vulnerability assessments” and “penetration tests,” and sometimes the two are combined and colloquially termed as a “VAPT” exercise. It’s important to understand that VAPT is the collective of two separate exercises that have distinctly varied intentions and outcomes. Let’s drill in a bit further.
According to the National Institute of Standards and Technology (NIST), a vulnerability assessment is an exercise aimed at uncovering security weaknesses in a system either manually or, more prevalently, using automated tools or scanners. By extension, the desired outcome from a vulnerability assessment is to ascertain the presence of potential access points in an environment, irrespective of exploitability. This exercise is especially useful to flag system weaknesses such as unpatched components or vulnerabilities that need to be identified and fixed irrespective of its ease of exploit or impact to the affected system or application.
A penetration test, as EC Council defines it, is often a fixed-time exercise with guard-rails in place to ascertain exploitability of vulnerabilities in a system. Most penetration testing approaches would factor in some form of vulnerability assessment followed by a penetration test, since a pen test, in theory, requires the presence of a vulnerability.
Therefore, while a vulnerability assessment can be an independent exercise, a penetration test is dependent on a vulnerability assessment to identify the exploitable weaknesses in the environment.
In layperson’s terms, the classic example is this: a vulnerability assessment rattles the doorknob to see if the room is unlocked. A penetration test uses the unlocked doors to enter the room, thus demonstrating an attacker’s ability to successfully break in. Naturally, this is guided by the client and is done in an authorized and controlled way.
In the context of the definitions provided above, we may now begin to appreciate the areas of applicability and specific outcomes from either of these types of testing. For example, an automated vulnerability scan is sometimes run upon mature environments, where the chances of finding serious vulnerabilities may be limited, or in critical environments, where even a low-priority finding could result in a state of non-compliance. A subset of vulnerability assessments are automated vulnerability scans run via commercial scanner services or platforms on network infrastructure or application components. Examples also include SAST, DAST and SCA scans in the context of secure coding practices. Most vulnerability scan results require some sort of a triage functionality—either to ascertain false positives (or true negatives) or to qualify the severity of the vulnerability.
That qualification, in a broad sense, has all the elements required to perform a penetration test.
A penetration testing exercise, in the practical sense, involves variables that introduce subjectivity in results. Such variables might include factors like the test environment, approach/methodology used during the test, available duration for testing and finally the perspective of the tester. When preceded by a formal vulnerability assessment, the penetration testing exercise is also heavily influenced by the quality of the findings from the vulnerability assessment.
The Gradients of a Security Assessment Program
While every type of testing is intended to identify significant findings, there are cases in which certain issues don't appear in the final report. Additionally, some of those missed issues wind up being uncovered by other teams due to the fluidity of one or more of the mentioned variables.
Organizations can often achieve counterbalance by augmenting pen testing exercises through other gradients of vulnerability management such as internal tooling, vulnerability assessments, bug bounties, etc. The idea is to be able to unearth vulnerabilities across various stages of product development and prioritize remediation efforts based on the scope and nature of the assessment as well as the criticality of the asset (system) under consideration.
The gradient model also lends itself to managing vulnerabilities using a combination of internal teams and third-party providers. From an application security perspective, for example, the internal teams may run automated DAST scans (across deployment environments) to find and fix vulnerabilities early in the software development process, while third-party penetration testing partners can focus their efforts on unearthing issues that have fewer dependencies on automated scanners. Similarly, if the security teams have already passed an application release across internal security checks, including a penetration test, inviting security researchers as part of a bug bounty program to identify potential weaknesses that might have been missed or not considered makes most sense.
What to Look for in Third-Party Penetration Testing Providers
It is important that organizations engaging with independent, third-party pen testing providers establish clear expectations (preferably) with transparency on their current assessment programs. Some questions worth asking might include:
- Do they run internal scanners?
- Do daily check-ins get validated by a SAST/SCA platform?
- Does the organization perform internal penetration tests?
- Has the scope of assessment already been verified?
Responses to these questions would set clear expectations for the security consultants being engaged and provide them with valuable input to design their assessment test cases by allowing them to focus their efforts on unearthing the right type of issues.
Penetration Testing vs Vulnerability Assessment: Real World Example
Kroll was recently involved in scoping a pen testing program in which the client was very clear that the assessment needed to focus solely on business logic flaws. The organization already had internal checkpoints in place that had validated the presence of vulnerabilities that were more generic in nature, regressed from previous assessments or were commonly found by commercial DAST scanners. This clarity in expectations allowed the team conducting the engagement to plan the assessment with a “depth-first” approach, albeit within stipulated time frames.
Similarly, there have been engagements when the engineering teams do not have the bandwidth or infrastructure in place to have conducted any kind of assessment internally, thereby requiring the pen testing team to focus first on coverage and then on depth in subsequent assessments.
Are Both Vulnerability Assessments and Pen Tests Important?
At the core of an efficient enterprise vulnerability assessment program is understanding the scope and intention of testing and passing on that information to the involved parties. It is also critical to democratize assessments across both internal and external teams to have better visibility and control on both the scope and results of the assessment.
At the end of the day, both vulnerability assessments and penetration tests have their designated places in a robust vulnerability management program. Understanding what they mean, what they do and how they can be used together as part of a strategic approach can positively benefit an organization’s ability to harden defenses and reduce its security risk profile. Don’t let a case of mistaken identity or a semantic argument convince you otherwise.
