Static Application Vulnerability Testing: Binary Scanning vs Source Code Scanning
I had a meeting yesterday with a vendor who sells a SaaS solution for binary application vulnerability testing. They tell a very interesting story of a world where dynamic testing ("black box") takes place alongside static testing ("white box") to give you a full picture of your application security posture. They even combine the results with some e-Learning aspects so that developers can research the vulnerabilities in the same place they go to find them. In concept, this sounds fantastic, but I quickly turned into a skeptic and as I dug deeper into the details I'm not sure I like what I found.
I wanted to make sure I fully understood what was going on under the hood here so I started asking questions about the static testing and how it works. They've got a nice looking portal where you name your application, give it a version, assign it to a group of developers, and point it to your compiled code (WAR, EAR, JAR, etc). Once you upload your binaries, their system basically runs a disassembler on it to get it into assembly code. It's then at this level that they start looking for vulnerabilities. They said that this process takes about 3 days initially and then maybe 2 days after the first time because they are able to re-use some data about your application. Once complete, they say they are able to provide you a report detailing your vulnerabilities and how to fix them.
The thing that immediately struck me as worth noting here was the 2-3 day turnaround. This means that our developers would need to wait a fairly substantial amount of time before getting any feedback on the vulnerability status of their code. In a world full of Agile development, 2-3 days is a lifetime. Compare that to static source code testing where you get actionable results at compile time. The edge here definitely goes to source code testing as I believe most people would prefer the near-instant gratification.
The next thing worth noting was that they are taking binary files and disassembling them in order to find vulnerabilities. This lends itself to one major issue which is how can you determine with any accuracy the line number of a particular vulnerability written in let's say Java from assembly code generated by disassembling the binaries. By default, it's simply not possible. This vendor claimed that they can by adding in some debug strings at compile time, but even then I'd contend that you're not going to get much. I'm guessing they have some heuristics that are able to tell what function generated a set of assembly code, but I'm extremely skeptical that they can do anything with variable names, custom code functions, etc. I've seen some source code scanners, on the other hand, that not only tell you what line of code is affected, but are able to give you an entire list of parameters that have been consequently affected by that vulnerability. The edge here definitely goes to source code testing.
The main benefit that I can see with binary testing vs source code testing is that we can test code that we didn't write. Things like APIs, third-party applications, open source, etc are all things that we now have visibility into. The only problem here is that while we now can see the vulnerabilities in this software, they are unfortunately all things that we can't directly influence change in, unless we want to send our developers off to work on somebody else's software. I'd argue that scanning for vulnerabilities in that type of code is their responsibility, not ours. Granted, it'd be nice to have validation that there aren't vulnerabilities there that we're exposing ourselves to by uptaking it, but in all honesty are we really going to take the time to scan somebody else's work? Probably not. The edge here goes to binary testing with the caveat being that it's in something that I frankly don't care as much about.
This isn't the complete list of pros and cons by any means. It's just me voicing in writing some concerns that I had about the technology while talking to this particular vendor. In my opinion, the benefits of doing source code testing far outweigh any benefits that we could get from testing compiled binary files. What do you think about the benefits of one versus the other? I'd certainly love for someone to try to change my mind here and show me where the real value lies in binary testing.
Beware the Deceptive SLA, My Friend
We're trying to come to an agreement with a SaaS vendor about performance and availability service level agreements (SLAs). I discussed this topic some in my previous "SaaS Headaches" post. I thought it would be instructive to show people the standard kind of "defense in depth" that suppliers can have to protect against being held responsible for what they host for you.
We've been working on a deal with one specific supplier. As part of it, they'll be hosting some images for our site. There's a business team primarily responsible for evaluating their functionality etc., we're just in the mix as the faithful watchdogs of performance and availability for our site.
Round 1 - "What are these SLAs you speak of?" The vendor offers no SLA. "Unacceptable," we tell the project team. They fret about having to worry about that along with the 100 other details of coming to an agreement with the supplier, but duly go back and squeeze them. It takes a couple squeezes because the supplier likes to forget about this topic - send a list of five questions with one of them being "SLA," you get four answers back, ignoring the SLA question.
Round 2 - "Oh, you said 'SLA'! Oh, sure, we have one of those." We read the SLA and it only commits to their main host being pingable. Our service could be completely down, and it doesn't speak to that. Back to our project team, who now between the business users, procurement agent, and legal guy need more urging to lean on the supplier. The supplier plays dumb for a while, and then...
Cloud Headaches?
The industry is abuzz with people who are freaked out about the outages that Amazon and other cloud vendors have had. "Amazon S3 Crash Raises Doubts Among Cloud Customers," says InformationWeek!
This is because people are going into cloud computing with retardedly high expectations. This year at Velocity, Interop, etc. I've seen people just totally in love with cloud computing - Amazon's specifically but in general as well. And it's a good concept for certain applications. However, it is a computing system just like every other computing system devised previously by man. And it has, and will have, problems.
Whether you are using in house systems, or a SaaS vendor, or building "in the cloud," you have the same general concerns. Am I monitoring my systems? What is my SLA? What is my recourse if my system is not hitting it? What's my DR plan?
Cloud computing is also being called "PaaS," or Platform as a Service. It's a special case of SaaS. And if you're a company relying on it, when you contract with a SaaS vendor you get SLAs established and figure out what the remedy is if they breach it. If you are going into a relationship where you are just paying money for a cloud VM, storage, etc. and there is no enforceable SLA in the relationship, then you need to build the risk of likely and unremediable outages into your business plan.
I hate to break it to you, but the IT people working at Amazon, Google, etc. are not all that smarter than the IT people working with you. So an unjustified faith in a SaaS or cloud vendor - "Oh, it's Amazon, I'm sure they'll never have an outage of any sort - their entire system or localized to my part - and if they do I'm sure the $100/month I'm paying them will cause them to give a damn about me" - is unreasonable on its face.
Clouds and cloud vendors are a good innovation. But they're like every other computing innovation and vendor selling it to you. They'll have bugs and failures. But treating them differently is a failure on your part, not theirs.
SaaS Headaches
There's a lot of promise in the new SaaS (software as a service; what used to be called ASPs, or Application Service providers, till Microsoft crapped all over that acronym) and newer PaaS (platform as a service) spaces (and look for a steady stream of new "aaS"es to come). However, there are a lot of gotchas in signing on with a SaaS vendor. You'd like to be able to believe that they have decent performance, uptime, security, etc., especially after the tell you "Oh, all kinds of big companies use us; Dell, IBM..." This is exacerbated by SaaS often being an "end run" around IT in the enterprise, so naive users can get sold a bill of goods without proper technical oversight. SaaS is a big buzzword now, and there are a lot of startups springing up that do not necessarily have experience running large scale sites. Think about how many MMORPG games still get scuttled due to poor operational performance. SaaS is the same.
Here's some things to keep in mind when selecting a SaaS vendor, laced with real life horror stories from our experiences.
1. Performance/Availability. Set a hard performance/availability SLA in the contract. Many vendors won't even have an SLA clause, or they'll have one that says "99.9% uptime!" without any remedy clause for what if they don't hit that. You want a clear SLA with a clear measurement method and clear "money back" if they don't hit it. We use a 2 second global performance SLA as measured by a Keynote Global 35 monitor. But the SLA isn't the whole story - you are counting on these people to accomplish your goals.