By: Josh

Josh — Thu, 25 Sep 2008 12:24:43 +0000

The method laid out by Jacob and Taylor in this presentation for comparing source code analysis to penetration testing to application firewalls is a pretty good way to compare different things in an organized and even manner. NI actually does something fairly similar when comparing different vendors with similar tools and we call it a “Supplier Selection Matrix (SSM)” or a “Vendor Report Card”. This is just an Excel spreadsheet with the rows being your evaluation criteria and columns being the vendors you want to compare. We break the evaluation criteria into administration, management, market position, usability, architecture, extensability categories and will add additional categories for additional functionality that we want to evaluate based on the specific type of vendor we are evaluating. For example, if we are evaluating a vulnerability detection tool, we probably want an additional category for “vulnerability detection” features that accounts for the number of vulnerabilities detected, quality of vulnerabilities detected, and accuracy of vulnerabilities detected. The last step is to provide a 1-10 weight for each evaluation criteria category for how important it is to you. Then, give each tool a 1-5 rating in each category and calculate the weighted average (cat1 rating x weight1 + cat2 rating + weight2 + …)/(weight1 + weight2 + …) for each tool. The number that you come out with is the grade for that tool. 0-0.999=F, 1-1.999=D, 2-2.999=C, 3-3.999=B, 4-5=A.

Comments on: Mastering PCI Section 6.6 – OWASP AppSec NYC 2008

By: Josh