This presentation is by Jacob West in the Security Research Group and Taylor McKinsley in Product Marketing from Fortify software.  I’d like to note that Fortify is a developer of a source code analysis tool and so this presentation may have a bias towards source code analysis tools.

56% of organizations fail PCI section 6.  Poorly coded web applications leading to SQL injection vulnerabilities is one of hte top five reasons for a PCI audit failure.  Section 6 is becoming a bigger problem: #9 in 2006 reason for failure, #2 in 2007.

PCI Section 6 has to do with guidelines to “Develop and maintain secure systems and applications”.  Section 6.6 reads “Ensure that all web-facing applications are protected against known attacks by either of the following methods: Having all custom application code reviwed for common vulnerabilities by an organization that specializes in web application secure” or by using a web application firewall.  Further clarifications say that automated tools are acceptable, web application penetration testing is allowed, and vulnerability assessments can be performed by an internal team.

Comparing Apples, Oranges, and Watermelons

  • Setup: Source code analysis (+2) is good because it works on existing hardware, but must live where your source code lives.  Penetration testing (+3) is good because you only need one to assess everything and works on existing hardware, but needs to talk to a running program.  Application firewall (+1)is good because it lives on the network, but you must model program behavior.
  • Optimization: Source code analysis (+2) is good because you can specify generic antipatterns in code, but you must understand vulnerability in detail.  Penetration testing (+2) is good because tests are attacks, but you must successfully attack your application.  Application firewalls (+1) are good because they share configuration across programs, but must differentiate good from bad.
  • Performance: Source code analysis (+3) is good because it simulates all application states and is non-production, but scales with build time and not the number of tests.  Penetration testing (+2) is good because you get incremental results and is non-production, but you must exercise each application state.  Application firewall (+1) is good because it’s a stand-alone device and scales with $$$, but impacts production performance and scales with $$$.
  • Human resources: Source code analysis (+1) is good because it enables security in development and reports a root cause, but makes auditors better and does not replace them.  Penetration testing (+2) is good because it is highly automatable, but reports symptoms and not the root cause.  Application firewall (+2) is good because once it’s configured it functions largely unattended, but requires extensive and ongoing configuration.
  • Security know-how: Source code analysis (+3) is good because it gives code-level details to an auditor, but you must understand security-relevant behavior of APIs.  Penetration testing (+1) is good because it automates hacks, but a hacker is required to measure success and optimize.  Application firewall (+2) is good because it identifies common attacks out of the box and is a community effort, but a hacker is required to measure success and customize.
  • Development expertise: Source code analysis (+1) is good because it focuses attention on relevant code, but you must understand code-level program behavior.  Penetration testing (+2) is good because basic attacks ignore internals, but advanced attacks require internal knowledge.  Application firewalls (+2) are good because they live on the network, but you must understand the program to tell good from bad.
  • False positives: Source code analysis (+1) is good because it gives auditors details to verify issues, but reports impossible application states.  Penetration testing (+2) is good because results come with reproduction steps, but it is difficult to oracle some bugs.  Application firewalls (+1) are good because it is attacks instead of vulnerabilities, but there is an evolving definition of valid behavior.
  • False negatives: Source code analysis (+3) is good because it simulates all program states and models the full program, but it must be told what to look for.  Penetration testing (+1) is good because it is good at finding what hackers find, but is difficult to oracle some bugs and has missed coverage.  Application firewalls (+1) are good because it uses attacks instead of vulnerabilities, but there is an evolving attack landscape.
  • Technology support: Source code analysis (+2) is good because parsing is separable from the analysis and is interface-neutral, but it must adapt to new program paradigms.  Penetration testing (+2) is good because it is independent from program paradigms, but is tied to protocols and is limited to network interfaces.  Application firewalls (+2) are good because they are independent from program paradigms, but are tied to protocols and are limited to network interfaces.

Working Towards a Solution

  • Assessment: Proving the problem or meeting the regulatory requirement.  Recurring cost that does not “fix” anything
  • Remediation: Fixing security issues found during assessments.  Lowering business risk at a single point in time.
  • Prevention: Get security right hte first time.  Minimizing business risk systematically.

Do your own comparison and fill out the scorecard yourself (presenters ratings are noted in parentheses above).

Taylor did interviews with three companies to get their experiences deploying each (source code analysis, penetration testing, and application firewall) and had them evaluate based on the nine criteria both before and after buying.  Not going to list each company’s results in the blog, but it was just a basic table with each criteria and a number rating for both before purchase and after deployment.  To sum it up, Source Code Analysis was a 14 rating before purchase and a 17 rating after deployment.  Penetration testing was a 21 rating before purchase and a 21 rating after deployment.  Application firewalls were a 21 rating before purchase and a 16 rating after deployment.  It seems like the first organization had a large amount of developers and that factored into their decision to purchase a source code analysis tool.  The second organization had a far fewer number of developers and was more of an IT shop and chose the penetration testing tool.  The last organization was a smaller shop in general (still fairly large) and went with the WAF because they wanted something they could just put in place and manage.

Analysis: All three solutions required more effort than expected.  All three solutions produce reasonably accurate results.  Varying levels of expertise needed.

How do you demonstrate that your application is protected against known attacks?

  • Verification that the application was analyzed
  • A report showing no critical security issues identified
  • Document showing how the tool fits into your architecture

How do you show that the user is appropriately trained?

  • Document explaining prior experience or an informal interview

How do you show that you have configured the tool appropriately?

  • Document explaining how the tool was configured and what new rules had to be added.

Summary: PCI section 6 is evolving to become increasingly precise.  Compare technologies in your environment along nine criteria.  Demonstrating compliance is an art, not a science.