Static Application Vulnerability Testing: Binary Scanning vs Source Code Scanning
I had a meeting yesterday with a vendor who sells a SaaS solution for binary application vulnerability testing. They tell a very interesting story of a world where dynamic testing ("black box") takes place alongside static testing ("white box") to give you a full picture of your application security posture. They even combine the results with some e-Learning aspects so that developers can research the vulnerabilities in the same place they go to find them. In concept, this sounds fantastic, but I quickly turned into a skeptic and as I dug deeper into the details I'm not sure I like what I found.
I wanted to make sure I fully understood what was going on under the hood here so I started asking questions about the static testing and how it works. They've got a nice looking portal where you name your application, give it a version, assign it to a group of developers, and point it to your compiled code (WAR, EAR, JAR, etc). Once you upload your binaries, their system basically runs a disassembler on it to get it into assembly code. It's then at this level that they start looking for vulnerabilities. They said that this process takes about 3 days initially and then maybe 2 days after the first time because they are able to re-use some data about your application. Once complete, they say they are able to provide you a report detailing your vulnerabilities and how to fix them.
The thing that immediately struck me as worth noting here was the 2-3 day turnaround. This means that our developers would need to wait a fairly substantial amount of time before getting any feedback on the vulnerability status of their code. In a world full of Agile development, 2-3 days is a lifetime. Compare that to static source code testing where you get actionable results at compile time. The edge here definitely goes to source code testing as I believe most people would prefer the near-instant gratification.
The next thing worth noting was that they are taking binary files and disassembling them in order to find vulnerabilities. This lends itself to one major issue which is how can you determine with any accuracy the line number of a particular vulnerability written in let's say Java from assembly code generated by disassembling the binaries. By default, it's simply not possible. This vendor claimed that they can by adding in some debug strings at compile time, but even then I'd contend that you're not going to get much. I'm guessing they have some heuristics that are able to tell what function generated a set of assembly code, but I'm extremely skeptical that they can do anything with variable names, custom code functions, etc. I've seen some source code scanners, on the other hand, that not only tell you what line of code is affected, but are able to give you an entire list of parameters that have been consequently affected by that vulnerability. The edge here definitely goes to source code testing.
The main benefit that I can see with binary testing vs source code testing is that we can test code that we didn't write. Things like APIs, third-party applications, open source, etc are all things that we now have visibility into. The only problem here is that while we now can see the vulnerabilities in this software, they are unfortunately all things that we can't directly influence change in, unless we want to send our developers off to work on somebody else's software. I'd argue that scanning for vulnerabilities in that type of code is their responsibility, not ours. Granted, it'd be nice to have validation that there aren't vulnerabilities there that we're exposing ourselves to by uptaking it, but in all honesty are we really going to take the time to scan somebody else's work? Probably not. The edge here goes to binary testing with the caveat being that it's in something that I frankly don't care as much about.
This isn't the complete list of pros and cons by any means. It's just me voicing in writing some concerns that I had about the technology while talking to this particular vendor. In my opinion, the benefits of doing source code testing far outweigh any benefits that we could get from testing compiled binary files. What do you think about the benefits of one versus the other? I'd certainly love for someone to try to change my mind here and show me where the real value lies in binary testing.
Assessing Your Web App Manually Without Hacking It
After giving my presentation on "Using Proxies to Secure Applications and More" at the TRISC 2009 conference, I decided to attend the presentation by Robert "RSnake" Hansen and Rob MacDougal entitled "Assessing Your Web App Manually Without Hacking It". The gist of this presentation was that with a few simple tools (Web Developer Toolbar, NoScript, you web browser) you can spend about an hour looking at the characteristics of a web application in order to determine what types and how many vulnerabilities it may have. My notes on the presentation are below:
Mastering PCI Section 6.6 – OWASP AppSec NYC 2008
This presentation is by Jacob West in the Security Research Group and Taylor McKinsley in Product Marketing from Fortify software. I'd like to note that Fortify is a developer of a source code analysis tool and so this presentation may have a bias towards source code analysis tools.
56% of organizations fail PCI section 6. Poorly coded web applications leading to SQL injection vulnerabilities is one of hte top five reasons for a PCI audit failure. Section 6 is becoming a bigger problem: #9 in 2006 reason for failure, #2 in 2007.
PCI Section 6 has to do with guidelines to "Develop and maintain secure systems and applications". Section 6.6 reads "Ensure that all web-facing applications are protected against known attacks by either of the following methods: Having all custom application code reviwed for common vulnerabilities by an organization that specializes in web application secure" or by using a web application firewall. Further clarifications say that automated tools are acceptable, web application penetration testing is allowed, and vulnerability assessments can be performed by an internal team.
Comparing Apples, Oranges, and Watermelons
- Setup: Source code analysis (+2) is good because it works on existing hardware, but must live where your source code lives. Penetration testing (+3) is good because you only need one to assess everything and works on existing hardware, but needs to talk to a running program. Application firewall (+1)is good because it lives on the network, but you must model program behavior.
- Optimization: Source code analysis (+2) is good because you can specify generic antipatterns in code, but you must understand vulnerability in detail. Penetration testing (+2) is good because tests are attacks, but you must successfully attack your application. Application firewalls (+1) are good because they share configuration across programs, but must differentiate good from bad.
- Performance: Source code analysis (+3) is good because it simulates all application states and is non-production, but scales with build time and not the number of tests. Penetration testing (+2) is good because you get incremental results and is non-production, but you must exercise each application state. Application firewall (+1) is good because it's a stand-alone device and scales with $$$, but impacts production performance and scales with $$$.
- Human resources: Source code analysis (+1) is good because it enables security in development and reports a root cause, but makes auditors better and does not replace them. Penetration testing (+2) is good because it is highly automatable, but reports symptoms and not the root cause. Application firewall (+2) is good because once it's configured it functions largely unattended, but requires extensive and ongoing configuration.
- Security know-how: Source code analysis (+3) is good because it gives code-level details to an auditor, but you must understand security-relevant behavior of APIs. Penetration testing (+1) is good because it automates hacks, but a hacker is required to measure success and optimize. Application firewall (+2) is good because it identifies common attacks out of the box and is a community effort, but a hacker is required to measure success and customize.
- Development expertise: Source code analysis (+1) is good because it focuses attention on relevant code, but you must understand code-level program behavior. Penetration testing (+2) is good because basic attacks ignore internals, but advanced attacks require internal knowledge. Application firewalls (+2) are good because they live on the network, but you must understand the program to tell good from bad.
- False positives: Source code analysis (+1) is good because it gives auditors details to verify issues, but reports impossible application states. Penetration testing (+2) is good because results come with reproduction steps, but it is difficult to oracle some bugs. Application firewalls (+1) are good because it is attacks instead of vulnerabilities, but there is an evolving definition of valid behavior.
- False negatives: Source code analysis (+3) is good because it simulates all program states and models the full program, but it must be told what to look for. Penetration testing (+1) is good because it is good at finding what hackers find, but is difficult to oracle some bugs and has missed coverage. Application firewalls (+1) are good because it uses attacks instead of vulnerabilities, but there is an evolving attack landscape.
- Technology support: Source code analysis (+2) is good because parsing is separable from the analysis and is interface-neutral, but it must adapt to new program paradigms. Penetration testing (+2) is good because it is independent from program paradigms, but is tied to protocols and is limited to network interfaces. Application firewalls (+2) are good because they are independent from program paradigms, but are tied to protocols and are limited to network interfaces.
Working Towards a Solution
- Assessment: Proving the problem or meeting the regulatory requirement. Recurring cost that does not "fix" anything
- Remediation: Fixing security issues found during assessments. Lowering business risk at a single point in time.
- Prevention: Get security right hte first time. Minimizing business risk systematically.
Do your own comparison and fill out the scorecard yourself (presenters ratings are noted in parentheses above).
Taylor did interviews with three companies to get their experiences deploying each (source code analysis, penetration testing, and application firewall) and had them evaluate based on the nine criteria both before and after buying. Not going to list each company's results in the blog, but it was just a basic table with each criteria and a number rating for both before purchase and after deployment. To sum it up, Source Code Analysis was a 14 rating before purchase and a 17 rating after deployment. Penetration testing was a 21 rating before purchase and a 21 rating after deployment. Application firewalls were a 21 rating before purchase and a 16 rating after deployment. It seems like the first organization had a large amount of developers and that factored into their decision to purchase a source code analysis tool. The second organization had a far fewer number of developers and was more of an IT shop and chose the penetration testing tool. The last organization was a smaller shop in general (still fairly large) and went with the WAF because they wanted something they could just put in place and manage.
Analysis: All three solutions required more effort than expected. All three solutions produce reasonably accurate results. Varying levels of expertise needed.
How do you demonstrate that your application is protected against known attacks?
- Verification that the application was analyzed
- A report showing no critical security issues identified
- Document showing how the tool fits into your architecture
How do you show that the user is appropriately trained?
- Document explaining prior experience or an informal interview
How do you show that you have configured the tool appropriately?
- Document explaining how the tool was configured and what new rules had to be added.
Summary: PCI section 6 is evolving to become increasingly precise. Compare technologies in your environment along nine criteria. Demonstrating compliance is an art, not a science.