Static Application Vulnerability Testing: Binary Scanning vs Source Code Scanning
I had a meeting yesterday with a vendor who sells a SaaS solution for binary application vulnerability testing. They tell a very interesting story of a world where dynamic testing ("black box") takes place alongside static testing ("white box") to give you a full picture of your application security posture. They even combine the results with some e-Learning aspects so that developers can research the vulnerabilities in the same place they go to find them. In concept, this sounds fantastic, but I quickly turned into a skeptic and as I dug deeper into the details I'm not sure I like what I found.
I wanted to make sure I fully understood what was going on under the hood here so I started asking questions about the static testing and how it works. They've got a nice looking portal where you name your application, give it a version, assign it to a group of developers, and point it to your compiled code (WAR, EAR, JAR, etc). Once you upload your binaries, their system basically runs a disassembler on it to get it into assembly code. It's then at this level that they start looking for vulnerabilities. They said that this process takes about 3 days initially and then maybe 2 days after the first time because they are able to re-use some data about your application. Once complete, they say they are able to provide you a report detailing your vulnerabilities and how to fix them.
The thing that immediately struck me as worth noting here was the 2-3 day turnaround. This means that our developers would need to wait a fairly substantial amount of time before getting any feedback on the vulnerability status of their code. In a world full of Agile development, 2-3 days is a lifetime. Compare that to static source code testing where you get actionable results at compile time. The edge here definitely goes to source code testing as I believe most people would prefer the near-instant gratification.
The next thing worth noting was that they are taking binary files and disassembling them in order to find vulnerabilities. This lends itself to one major issue which is how can you determine with any accuracy the line number of a particular vulnerability written in let's say Java from assembly code generated by disassembling the binaries. By default, it's simply not possible. This vendor claimed that they can by adding in some debug strings at compile time, but even then I'd contend that you're not going to get much. I'm guessing they have some heuristics that are able to tell what function generated a set of assembly code, but I'm extremely skeptical that they can do anything with variable names, custom code functions, etc. I've seen some source code scanners, on the other hand, that not only tell you what line of code is affected, but are able to give you an entire list of parameters that have been consequently affected by that vulnerability. The edge here definitely goes to source code testing.
The main benefit that I can see with binary testing vs source code testing is that we can test code that we didn't write. Things like APIs, third-party applications, open source, etc are all things that we now have visibility into. The only problem here is that while we now can see the vulnerabilities in this software, they are unfortunately all things that we can't directly influence change in, unless we want to send our developers off to work on somebody else's software. I'd argue that scanning for vulnerabilities in that type of code is their responsibility, not ours. Granted, it'd be nice to have validation that there aren't vulnerabilities there that we're exposing ourselves to by uptaking it, but in all honesty are we really going to take the time to scan somebody else's work? Probably not. The edge here goes to binary testing with the caveat being that it's in something that I frankly don't care as much about.
This isn't the complete list of pros and cons by any means. It's just me voicing in writing some concerns that I had about the technology while talking to this particular vendor. In my opinion, the benefits of doing source code testing far outweigh any benefits that we could get from testing compiled binary files. What do you think about the benefits of one versus the other? I'd certainly love for someone to try to change my mind here and show me where the real value lies in binary testing.
Oracle + BEA Update
A year ago I wrote about Oracle's plan on how to combine BEA Weblogic and OAS. A long time went by before any more information appeared - we met with our Oracle reps last week to figure out what the deal is. The answer wasn't much more clear than it was way back last year. They do certainly want some kind of money to "upgrade" but it seems poorly thought through.
OAS came in various versions - Java, Standard, Standard One, Enterprise, and then the SOA Suite versions. The new BEA, now "Fusion Middleware 11g" comes in different versions as well.
- WLS Standard
- WLS Enterprise - adds clustering, costs double
- WLS Suite - adds Coherence, Enterprise Manager, and JRockit realtime, costs quadruple
But they can't tell us what OAS product maps to what FMW version.
There is also an oddly stripped down "Basic" edition which noted as being a free upgrade from OAS SE but it strips out a lot of JMS and WS stuff; there's an entire slide of stuff that gets stripped out and it's hard to say if this would be feasible for us.
As for SOA Suite, "We totally just don't know."
Come on Oracle, you've had a year to get this put together. It's pretty simple, there's not all that many older and newer products. I suspect they're being vague so they can feel out how much $$ they can get out of people for the upgrade. Hate to break it to you guys - the answer is $0. We didn't pay for OAS upgrades before this, we just paid you the generous 22% a year maintenance that got you your 51% profit margin this year. If you're retiring OAS for BEA in all but name, we expect to get the equivalent functionality for our continued 22%.
Oracle has two (well, three) clear to dos.
1. Figure out what BEA product bundles give functionality equivalent to old OAS bundles
2. Give those to support-paying customers
3. Profit. You're making plenty without trying to upcharge customers. Don't try it.
Customizing Apache Error Codes By URL
I've had a couple of discussions lately about customized Apache error pages that prompted me to do a little bit of research on it. What I've come up with is somewhat interesting so I thought I'd share it with everyone. First, it is not technically possible to tell Apache to serve up a different error page for image content than for html content than for php content since the only command Apache accepts for this is of the "ErrorDocument error-code document" format. That said, if you allow .htaccess overrides on a particular directory, then you can specify your ErrorDocument directive in there as well; overriding the default error handling specified in the httpd.conf file. An example:
In my httpd.conf file I have all 404's going to errorpage.cgi with the following line:
ErrorDocument 404 /cgi-bin/errorpage.cgi
I'm a good little code monkey and put all of my images in a /images directory under the DirectoryRoot. By default, if I were to hit a non-existent image in that directory, I would get the default error message defined in the httpd.conf file. If that image were referenced in an html page that I hit, I now download the html page plus the errorpage.cgi page for the bad image reference, introducing one whole page's worth of additional overhead.
But since I was a good code monkey and put all of my images in a /images directory, the fix for this is really simple. I create a .htaccess file inside of my /images directory and add the following line to it:
ErrorDocument 404 "404 - Image does not exist <-- Note: No end quote is intentional
Now, if I hit http://www.mysite.com/badpage.html I get the errorpage.cgi page, but if I hit http://www.mysite.com/images/badimage.jpg I get a short and sweet message saying "404 - Image does not exist". I haven't tested this yet to see how it works when you are using something like mod_oc4j to send certain URLs to an application server, but it's possible that this could work there too if Apache checks for existing static URLs before passing requests to the app server. Further testing could be useful there.
So there you have it. I can't tell Apache to serve up different error pages based on the URL or file type, but if I'm diligent about putting different files under different directories, I can effectively do the same thing using .htaccess files. Woot!
Google Chrome Hates You (Error 320)
The 1.0 release of Google Chrome has everyone abuzz. Here at NI, loads of people are adopting it. Shortly after it went gold, we started to hear from users that they were having problems with our internal collaboration solution, based on the Atlassian Confluence wiki product. They'd hit a page and get a terse error, which if you clicked on "More Details" you got the slightly more helpful, or at least Googleable, string "Error 320 (net::ERR_INVALID_RESPONSE): Unknown error."
At first, it seemed like if people reloaded or cleared cache the problem went away. It turned out this wasn't true - we have two load balanced servers in a cluster serving this site. One server worked in Chrome and the other didn't; reloading or otherwise breaking persistence just got you the working server for a time. But both servers worked perfectly in IE and Firefox (every version we have lying around).
So we started researching. Both servers were as identical as we could make them. Was it a Confluence bug? No, we have phpBB on both servers and it showed the same behavior - so it looked like an Apache level problem.
Sure enough, I looked in the logs. The error didn't generate an Apache error, it was still considered a 200 OK response, but when I compared the log strings the box that Chrome was erroring on showed that the cookie wasn't being passed up; that field was blank (it was populated with the cookie value on the other box and on both boxes when hit in IE/Firefox). Both boxes had an identically compiled Apache 2.0.61. I diffed all the config files- except for boxname and IP, no difference. The problem persisted for more than a week.
We did a graceful Apache restart for kicks - no effect. Desperate, we did a full Apache stop/start - and the problem disappeared! Not sure for how long. If it recurs, I'll take a packet trace and see if Chrome is just not sending the cookie, or sending it partially, or sending it and it's Apache jacking up... But it's strange there would be an Apache-end problem that only Chrome would experience.
I see a number of posts out there in the wide world about this issue; people have seen this Chrome behavior in YouTube, Lycos, etc. Mostly they think that reloading/clearing cache fixes it but I suspect that those services also have large load balanced clusters, and by luck of the draw they're just getting a "good" one.
Any other server admins out there having Chrome issues, and can confirm this? I'd be real interested in knowing what Web servers/versions it's affecting. And a packet trace of a "bad" hit would probably show the root cause. I suspect for some reason Chrome is partially sending the cookie or whatnot, choking the hit.
Using Proxies to Secure Applications and More
I've been really surprised that for as long as I've been active with OWASP, I've never seen a proxy presentation. After all, they are hugely beneficial in doing web application penetration testing and they're really not that difficult to use. Take TamperData for example. It's just a firefox plugin, but it does header, cookie, get, and post manipulation just as well as WebScarab. Or Google Ratproxy, which works in the background while you browse around QA'ing your web site and gives you a nice actionable report when you're done. I decided it was time to educate my peers on the awesomeness of proxies.
This past Tuesday I presented to a crowd of about 35 people at the Austin OWASP Meeting. The title of my presentation was "Using Proxies to Secure Applications and More". Since so many people came up to me afterward telling me what a great presentation it was and how they learned something they can take back to the office, I decided (with a little insistance from Ernest) that it was worth putting up on SlideShare and posting to the Web Admin Blog.
The presentation starts off with a brief description of what a proxy is. Then, I talked about the different types of proxies. Then, the bulk of the presentation was just me giving examples and demonstrating the various proxies. I included anonymizing proxies, reverse proxies, and intercepting proxies. While my slides can't substitue for the actual demo, I did try to include in them what tool I used for the demo. If you have any specific questions, please let me know. All that said, here's the presentation.
An Evaluation of Rapid7 NeXpose
I've been focusing a lot of my time lately on our PCI initiatives. One sub-topic that I've spent a particularly large amount of time on has been Requirement 11.2 which says that you need to have internal and external network vulnerability scans performed by a scan vendor qualified by PCI. We already employ one such tool, but I've been working to evaluate several other vulnerability scanning tools to see where our current tool is at in comparison. I'll post my evaluations of each of these tools in time, but for now I'll start with my evaluation of Rapid7 NeXpose.
Google Ratproxy
If you are responsible for developing or maintaining a website and haven't checked out Ratproxy yet, you're missing out. Before I start spouting off about just how cool and useful this tool is, I suppose I should first tell you what a proxy is. In a nutshell, a proxy is an application that runs local on your computer and intercepts requests and responses between your web browser and the web server. In almost all cases, the proxy has the ability to manipulate the conversation going on between the two. Things like modifying your cookies, changing POST and GET parameters, and finding hidden fields are made uber-easy with the assistance of a proxy.
Oracle + BEA = ?
We use Oracle Application Server as our Java app server at NI. Yeah, yeah, I'll wait till you stop laughing.
Why not JBoss or WebLogic or WebSphere? Well, a couple reasons. We made the decision five years ago, and JBoss wasn't solid then, and we needed J2EE support so plain Tomcat wasn't enough. And we're a huge Oracle shop and figured that if we were using the same app server on the Web and our ERP tiers there'd be leverage in terms of developer knowledge etc. Would we make that same decision today? I'm not sure about that (I can hear my team members shouting "hell no" over the cube walls). Although since we've also gone with Oracle's SOA suite for ESB and BPEL it would be harder to switch. But still tempting - Oracle has done a horrible job in getting their app server supported by other vendors. Every time we buy something and look at the supported app server section of their support matrix, and we ask "What about Oracle's OAS?" we get expressions of mixed horror and pity from the supplier. (I liked it when the Chinese technical guy from one eComm vendor we had in responded to this question with, "You know, the Tomcat is good, and free! Maybe you use that!")
Anyway, Oracle bought BEA a while back, which got keen interest from us. Stay with Oracle *and* use a good app server that other people support? Tempting! But Oracle's been farting around for six months without coming out with a statement on what this will mean for the products. Oracle's finally done a Webcast describing their strategy. Well, it's half marketing and a celebration of how many million dollars they have. But there's also a lot of product strategy in there. I'll sum it up for you because the damn webcast is nearly two hours long, and I don't want other people to have to waste that much time on it. Unless you like to hear someone go on about "strategic clarity" and "customer profiles," in which case this is two hours of bliss for you and you should watch it. Although I also had the stream break a bunch of times while watching. Who the heck uses RealPlayer any more? Anyway, here's a list of the interesting product facts from the Webcast. Some are marked with their timestamp if you want to fast forward to them and see more.
Quick Blogging Tip
All yesterday I was being annoyed by the need to write up my blog posts in another editor and paste them over into WordPress. You have to do that because composing text longer than about 3 sentences in a browser window is taking your life in your hands. But I discovered even in cutting and pasting from Wordpad you get bullshit formatting inserted that drives the TinyMCE editor crazy. And Notepad was giving me line break problems. (And it needs not be said that you should never ever paste from Word...)
But Robert cued me in to PureText, which is a little Windows addon that strips all formatting from text when you cut/paste it for you. By default you Windows+V instead of Control+V and voila, no crap. Yay!
Log Management for Dummies (aka Splunk)
Logs are one thing that I think are severely underutilized by most systems administrators. Most of us have taken the first step by actually logging the data, but neglect organizing it into any sort of manageable form. You'll probably argue that any hardcore *nix admin would be able to take the raw logs using grep, cut, awk, and a handful of other *nix power tools and turn it into consumable information, but that'll only get you so far.
Several months ago we evaluated a bunch of log management solutions with several goals in mind. We wanted a solution that was agile enough to be able to take in a wide variety of log formats as well as configuration files. It needed to shield sensitive information (passwords, credit card information, etc) from unauthorized users. It needed to provide us with a customizable interface where we could report on all of the log data it gathered. Lastly, it needed to provide our customers (developers) with the ability to self-service their own log files. After evaluating most of the major players in the log management arena, we found our ideal solution in a product called Splunk.
The first thing I noticed when evaluating Splunk was that they're not like everyone else. They're not trying to sell you some sort of logging appliance and they offer their software free for customers with 100 MB/day or less worth of logging. Getting Splunk installed was a breeze. You can have it up and running in minutes. It truly is Log Management for Dummies in that respect, but under the hood there is a highly configurable and customizable tool with an API that you could use to write your own applications to examine log files.
At this point I've mucked around with Splunk for a few months and our configuration is pretty intense. I've added in custom indexes to make my custom dashboards load faster. I've set Splunk up to create queryable metadata fields based on information in the logs. I've added filters for custom timestamps and auditing so we can tell if a log file has been modified. I've even set up a "deployment server" to distribute Splunk's configuration bundles to my various types of servers. This brings me to the one drawback of Splunk: Upgrading. Rumor has it that they are working on making it easier to upgrade from one version to the next, but for the time being it involves logging in to each server, stopping Splunk, upgrading the files, and restarting Splunk again. If you only had to upgrade every once in a while it would be fine, but they maintain a very active development team so I find myself constantly wanting to upgrade to get the latest bug fixes and features.
Other than that, Splunk does exactly what I tell it to do. It grabs all of our logs and presents them in a single intuitive interface. Think of it as a search engine for log and configuration files. Then, once I have the log data in front of me, I can create custom reports based on that data. If I want to, I can even alert based on information Splunk finds in my logs (send an e-mail to a developer every time their application throws an error message). Oh, did I mention that Splunk has a PCI Dashboard that you can install for free? Ask those other guys how much they charge for their PCI solution.
The next time you have some free time be sure to download Splunk and install it on one of your development servers. You won't be disappointed.