Static Application Vulnerability Testing: Binary Scanning vs Source Code Scanning
I had a meeting yesterday with a vendor who sells a SaaS solution for binary application vulnerability testing. They tell a very interesting story of a world where dynamic testing (“black box”) takes place alongside static testing (“white box”) to give you a full picture of your application security posture. They even combine the results with some e-Learning aspects so that developers can research the vulnerabilities in the same place they go to find them. In concept, this sounds fantastic, but I quickly turned into a skeptic and as I dug deeper into the details I’m not sure I like what I found.
I wanted to make sure I fully understood what was going on under the hood here so I started asking questions about the static testing and how it works. They’ve got a nice looking portal where you name your application, give it a version, assign it to a group of developers, and point it to your compiled code (WAR, EAR, JAR, etc). Once you upload your binaries, their system basically runs a disassembler on it to get it into assembly code. It’s then at this level that they start looking for vulnerabilities. They said that this process takes about 3 days initially and then maybe 2 days after the first time because they are able to re-use some data about your application. Once complete, they say they are able to provide you a report detailing your vulnerabilities and how to fix them.
The thing that immediately struck me as worth noting here was the 2-3 day turnaround. This means that our developers would need to wait a fairly substantial amount of time before getting any feedback on the vulnerability status of their code. In a world full of Agile development, 2-3 days is a lifetime. Compare that to static source code testing where you get actionable results at compile time. The edge here definitely goes to source code testing as I believe most people would prefer the near-instant gratification.
The next thing worth noting was that they are taking binary files and disassembling them in order to find vulnerabilities. This lends itself to one major issue which is how can you determine with any accuracy the line number of a particular vulnerability written in let’s say Java from assembly code generated by disassembling the binaries. By default, it’s simply not possible. This vendor claimed that they can by adding in some debug strings at compile time, but even then I’d contend that you’re not going to get much. I’m guessing they have some heuristics that are able to tell what function generated a set of assembly code, but I’m extremely skeptical that they can do anything with variable names, custom code functions, etc. I’ve seen some source code scanners, on the other hand, that not only tell you what line of code is affected, but are able to give you an entire list of parameters that have been consequently affected by that vulnerability. The edge here definitely goes to source code testing.
The main benefit that I can see with binary testing vs source code testing is that we can test code that we didn’t write. Things like APIs, third-party applications, open source, etc are all things that we now have visibility into. The only problem here is that while we now can see the vulnerabilities in this software, they are unfortunately all things that we can’t directly influence change in, unless we want to send our developers off to work on somebody else’s software. I’d argue that scanning for vulnerabilities in that type of code is their responsibility, not ours. Granted, it’d be nice to have validation that there aren’t vulnerabilities there that we’re exposing ourselves to by uptaking it, but in all honesty are we really going to take the time to scan somebody else’s work? Probably not. The edge here goes to binary testing with the caveat being that it’s in something that I frankly don’t care as much about.
This isn’t the complete list of pros and cons by any means. It’s just me voicing in writing some concerns that I had about the technology while talking to this particular vendor. In my opinion, the benefits of doing source code testing far outweigh any benefits that we could get from testing compiled binary files. What do you think about the benefits of one versus the other? I’d certainly love for someone to try to change my mind here and show me where the real value lies in binary testing.
July 22nd, 2010 at 4:30 pm
After talking with one of my developers this afternoon about this, a couple of corrections. First, it sounds like at least in the Java world that this service is likely decompiling binaries into Java byte code and not pure assembly though the sales guy did say “assembly”. Second, I’ll withdraw my cynicism of the capability for the service to determine the line number of a vulnerability from said assembly (or byte) code. It turns out that by adding a debug flag you are able to get down to a line number from the disassembled code. So if you’re using debug for production code, this would likely work as advertised. I guess we’ll call that category a draw.
August 22nd, 2010 at 10:12 pm
My thought is that static binary analysis is better utilized during the QA stage as this would allow feature/functionality testing and security testing to be performed concurrently with no impact to schedule. Additionally, the independent validation is a great way to ensure that developers are not bypassing security requirements.
When time is a factor, security takes a backseat to feature/functionality so developers will end up ditching anything that gets in the way, which means those expensive tools will not get used and the Total Cost of Ownership and value proposition are washed away.
Next, I would suggest reading the following documents for some additional perspective on static binary:
“What You See Is Not What You eXecute” http://www.cs.wisc.edu/wpis/papers/wysinwyx05.pdf
“Some Bad News and Some Good News” http://msdn.microsoft.com/en-us/library/ms972826
January 13th, 2011 at 6:42 am
I’ve worked for a SSCA (source) vendor and now my security firm partners with a SBCA (binary) vendor. A partial list of advantages using binary analysis include: testing the same code the attacker sees, testing the entire software supply chain and greater accuracy (less FPs and less FNs). If you are a developer that is knowledgeable about appsec, are doing agile and have an SSCA plug-in to your IDE, cool, but most developers are security novices at best and although you may not care about your company’s COTS vendor security, your management cares alot since PCI, HIPAA etc do and when your PII gets breached bad things happen and all employees feel the pain.