I wanted to make sure I fully understood what was going on under the hood here so I started asking questions about the static testing and how it works. They've got a nice looking portal where you name your application, give it a version, assign it to a group of developers, and point it to your compiled code (WAR, EAR, JAR, etc). Once you upload your binaries, their system basically runs a disassembler on it to get it into assembly code. It's then at this level that they start looking for vulnerabilities. They said that this process takes about 3 days initially and then maybe 2 days after the first time because they are able to re-use some data about your application. Once complete, they say they are able to provide you a report detailing your vulnerabilities and how to fix them.

The thing that immediately struck me as worth noting here was the 2-3 day turnaround. This means that our developers would need to wait a fairly substantial amount of time before getting any feedback on the vulnerability status of their code. In a world full of Agile development, 2-3 days is a lifetime. Compare that to static source code testing where you get actionable results at compile time. The edge here definitely goes to source code testing as I believe most people would prefer the near-instant gratification.

The next thing worth noting was that they are taking binary files and disassembling them in order to find vulnerabilities. This lends itself to one major issue which is how can you determine with any accuracy the line number of a particular vulnerability written in let's say Java from assembly code generated by disassembling the binaries. By default, it's simply not possible. This vendor claimed that they can by adding in some debug strings at compile time, but even then I'd contend that you're not going to get much. I'm guessing they have some heuristics that are able to tell what function generated a set of assembly code, but I'm extremely skeptical that they can do anything with variable names, custom code functions, etc. I've seen some source code scanners, on the other hand, that not only tell you what line of code is affected, but are able to give you an entire list of parameters that have been consequently affected by that vulnerability. The edge here definitely goes to source code testing.

The main benefit that I can see with binary testing vs source code testing is that we can test code that we didn't write. Things like APIs, third-party applications, open source, etc are all things that we now have visibility into. The only problem here is that while we now can see the vulnerabilities in this software, they are unfortunately all things that we can't directly influence change in, unless we want to send our developers off to work on somebody else's software. I'd argue that scanning for vulnerabilities in that type of code is their responsibility, not ours. Granted, it'd be nice to have validation that there aren't vulnerabilities there that we're exposing ourselves to by uptaking it, but in all honesty are we really going to take the time to scan somebody else's work? Probably not. The edge here goes to binary testing with the caveat being that it's in something that I frankly don't care as much about.

This isn't the complete list of pros and cons by any means. It's just me voicing in writing some concerns that I had about the technology while talking to this particular vendor. In my opinion, the benefits of doing source code testing far outweigh any benefits that we could get from testing compiled binary files. What do you think about the benefits of one versus the other? I'd certainly love for someone to try to change my mind here and show me where the real value lies in binary testing.

OAS came in various versions - Java, Standard, Standard One, Enterprise, and then the SOA Suite versions.  The new BEA, now "Fusion Middleware 11g" comes in different versions as well.

• WLS Standard
• WLS Enterprise - adds clustering, costs double
• WLS Suite - adds Coherence, Enterprise Manager, and JRockit realtime, costs quadruple

But they can't tell us what OAS product maps to what FMW version.

There is also an oddly stripped down "Basic" edition which noted as being a free upgrade from OAS SE but it strips out a lot of JMS and WS stuff; there's an entire slide of stuff that gets stripped out and it's hard to say if this would be feasible for us.

As for SOA Suite, "We totally just don't know."

Come on Oracle, you've had a year to get this put together.  It's pretty simple, there's not all that many older and newer products.  I suspect they're being vague so they can feel out how much $$ they can get out of people for the upgrade.  Hate to break it to you guys - the answer is $0.  We didn't pay for OAS upgrades before this, we just paid you the generous 22% a year maintenance that got you your 51% profit margin this year. If you're retiring OAS for BEA in all but name, we expect to get the equivalent functionality for our continued 22%.

Oracle has two (well, three) clear to dos.

1.  Figure out what BEA product bundles give functionality equivalent to old OAS bundles

2.  Give those to support-paying customers

3.  Profit.  You're making plenty without trying to upcharge customers.  Don't try it.

In my httpd.conf file I have all 404's going to errorpage.cgi with the following line:

ErrorDocument 404 /cgi-bin/errorpage.cgi

I'm a good little code monkey and put all of my images in a /images directory under the DirectoryRoot.  By default, if I were to hit a non-existent image in that directory, I would get the default error message defined in the httpd.conf file.  If that image were referenced in an html page that I hit, I now download the html page plus the errorpage.cgi page for the bad image reference, introducing one whole page's worth of additional overhead.

But since I was a good code monkey and put all of my images in a /images directory, the fix for this is really simple.  I create a .htaccess file inside of my /images directory and add the following line to it:

ErrorDocument 404 "404 - Image does not exist       <-- Note: No end quote is intentional

Now, if I hit http://www.mysite.com/badpage.html I get the errorpage.cgi page, but if I hit http://www.mysite.com/images/badimage.jpg I get a short and sweet message saying "404 - Image does not exist".  I haven't tested this yet to see how it works when you are using something like mod_oc4j to send certain URLs to an application server, but it's possible that this could work there too if Apache checks for existing static URLs before passing requests to the app server.  Further testing could be useful there.

So there you have it.  I can't tell Apache to serve up different error pages based on the URL or file type, but if I'm diligent about putting different files under different directories, I can effectively do the same thing using .htaccess files.  Woot!

At first, it seemed like if people reloaded or cleared cache the problem went away.  It turned out this wasn't true - we have two load balanced servers in a cluster serving this site.  One server worked in Chrome and the other didn't; reloading or otherwise breaking persistence just got you the working server for a time.  But both servers worked perfectly in IE and Firefox (every version we have lying around).

So we started researching.  Both servers were as identical as we could make them.  Was it a Confluence bug?  No, we have phpBB on both servers and it showed the same behavior - so it looked like an Apache level problem.

Sure enough, I looked in the logs.  The error didn't generate an Apache error, it was still considered a 200 OK response, but when I compared the log strings the box that Chrome was erroring on showed that the cookie wasn't being passed up; that field was blank (it was populated with the cookie value on the other box and on both boxes when hit in IE/Firefox).  Both boxes had an identically compiled Apache 2.0.61.  I diffed all the config files- except for boxname and IP, no difference.  The problem persisted for more than a week.

We did a graceful Apache restart for kicks - no effect.  Desperate, we did a full Apache stop/start - and the problem disappeared!  Not sure for how long.  If it recurs, I'll take a packet trace and see if Chrome is just not sending the cookie, or sending it partially, or sending it and it's Apache jacking up...  But it's strange there would be an Apache-end problem that only Chrome would experience.

I see a number of posts out there in the wide world about this issue; people have seen this Chrome behavior in YouTube, Lycos, etc.  Mostly they think that reloading/clearing cache fixes it but I suspect that those services also have large load balanced clusters, and by luck of the draw they're just getting a "good" one.

Any other server admins out there having Chrome issues, and can confirm this?  I'd be real interested in knowing what Web servers/versions it's affecting.  And a packet trace of a "bad" hit would probably show the root cause.  I suspect for some reason Chrome is partially sending the cookie or whatnot, choking the hit.

This past Tuesday I presented to a crowd of about 35 people at the Austin OWASP Meeting.  The title of my presentation was "Using Proxies to Secure Applications and More".  Since so many people came up to me afterward telling me what a great presentation it was and how they learned something they can take back to the office, I decided (with a little insistance from Ernest) that it was worth putting up on SlideShare and posting to the Web Admin Blog.

The presentation starts off with a brief description of what a proxy is.  Then, I talked about the different types of proxies.  Then, the bulk of the presentation was just me giving examples and demonstrating the various proxies.  I included anonymizing proxies, reverse proxies, and intercepting proxies.  While my slides can't substitue for the actual demo, I did try to include in them what tool I used for the demo.  If you have any specific questions, please let me know.  All that said, here's the presentation.

First off, I had never heard of the company before, but they were among the cheaper options of what I evaluated and apparently are doing some good things.  They got the SC Magazine recommendation for the month of August 2008 and they received a 5-star overall rating in said magazine.  The problem came as soon as I started talking to their salesperson.  From the start, the guy was coming off like a used car salesman asking questions like "What would it take to get you to buy by the end of this month?"  This was before I even saw an evaluation of the product.  From that point forward, I don't think a week went by where I didn't hear from the salesperson.  "How's the evaluation going?  Do you think you're going to buy?"  It got annoying very quickly.

The evaluation of the product went fairly smoothly.  My biggest gripe was that the company claimed that they did everything that Qualys does and more (they even forwarded me a press release on it), but ultimately failed to deliver on that promise when I found something rather large that Qualys finds and NeXpose does not.  To their benefit, Rapid7 had engineers and developers calling me and asking about the issue trying to get it into their system for me.  That was pretty cool, but ultimately they're getting paid to find these vulnerabilities for us.  You would think that they'd at least have all of the CVE items in their scanning tool.

My missing Qualys vulnerability aside, the NeXpose tool found plenty of issues.  This was both a positive and a negative since a lot of what it found had to do with a single vulnerability being exposed over and over through our site's faceted navigation.  It would have been nice if the scanner recognized that since it made the results look a lot worse than it actually is.  Also, when going through the results, I noticed quite a few false positives.  It seemed like most of these were due to the scanner just looking at a version number in a header instead of actually trying to test the vulnerability.  It found issues with Apache modules that we didn't even have enabled.

My favorite thing about the Rapid7 NeXpose vulnerability scanning tool was the reporting.  They provide some very good reports in there by default.  I found the "Remediation Plan Report" to be particularly interesting as it provided you with their suggested path to remediate our vulnerabilities most effeciently and effectively.  Was it better than the reporting that I've seen in other products?  Maybe, maybe not.

Anyway, my evaluation of Rapid7 NeXpose was coming to a close when I got a call from the salesperson last week.  It went something like this...

Salesperson: "Did you hear we got a recommendation from SC Magazine?  Yeah, things are busy here.  Your evaluation is taking longer than normal and I know you've had several issues with the product, do you think you're going to buy it?"

Me: "Nope, hadn't heard about the SC Magazine thing.  We've definitely worked through some issues.  Overall, the evaluation went well and I like the product.  Once I finish the other evaluations I'm working on, I'll let you know our decision."

Salesperson: "Well, with the amount of business we're getting with the SC Magazine article, I don't have time for you.  Feel free to call me back if you decide to buy our product, otherwise, good luck."

What do you say to that?  I got dumped by a salesperson, who I kept dropping hints to leave me alone to do my evaluation, because I was taking up too much of his time?  It's a little difficult to do an unbiased review after that, but I tried my best.

I don't claim to be an expert on proxies, but I have used several in the past including OWASP WebScarab and Paros. While both of these tools provide features as described above, Ratproxy takes a very different approach. You start up your proxy and tell the browser to pass requests through it. Simple enough. Then you just start surfing your website as though you were a regular user. In the background, Ratproxy is collecting all sort of useful information about the website. When you're done surfing the site, you run the report which comes out as a nice web page full of useful information about the site. It'll show you pages vulnerable to CSRF, XSS, and a host of other security vulnerabilities. It ranks then based on high, medium, and low impact and provides very good explanations of the issues it has found.

The Ratproxy tool has ports for Mac OS/X, Linux, and Cygwin (Windows). When I first tried to compile it in Cygwin, I had all sorts of error messages, but then I found this very help web page that told me exactly what libraries Cygwin was missing in order for me to compile it correctly. Part two of that article even goes on to tell you how to begin using Ratproxy.

To many, Web Application Security is a scary thing that takes a lot of time and effort to figure out how to do things right, but it doesn't have to be. You also don't have to pay an arm and a leg to do a decent security audit of your website. Start today by downloading Ratproxy and get a feel for how secure your site is without paying a dime.

Why not JBoss or WebLogic or WebSphere? Well, a couple reasons. We made the decision five years ago, and JBoss wasn't solid then, and we needed J2EE support so plain Tomcat wasn't enough. And we're a huge Oracle shop and figured that if we were using the same app server on the Web and our ERP tiers there'd be leverage in terms of developer knowledge etc.  Would we make that same decision today? I'm not sure about that (I can hear my team members shouting "hell no" over the cube walls).  Although since we've also gone with Oracle's SOA suite for ESB and BPEL it would be harder to switch. But still tempting - Oracle has done a horrible job in getting their app server supported by other vendors. Every time we buy something and look at the supported app server section of their support matrix, and we ask "What about Oracle's OAS?" we get expressions of mixed horror and pity from the supplier. (I liked it when the Chinese technical guy from one eComm vendor we had in responded to this question with, "You know, the Tomcat is good, and free! Maybe you use that!")

Anyway, Oracle bought BEA a while back, which got keen interest from us. Stay with Oracle *and* use a good app server that other people support?  Tempting!  But Oracle's been farting around for six months without coming out with a statement on what this will mean for the products. Oracle's finally done a Webcast describing their strategy. Well, it's half marketing and a celebration of how many million dollars they have. But there's also a lot of product strategy in there. I'll sum it up for you because the damn webcast is nearly two hours long, and I don't want other people to have to waste that much time on it. Unless you like to hear someone go on about "strategic clarity" and "customer profiles," in which case this is two hours of bliss for you and you should watch it.  Although I also had the stream break a bunch of times while watching.  Who the heck uses RealPlayer any more?  Anyway, here's a list of the interesting product facts from the Webcast.  Some are marked with their timestamp if you want to fast forward to them and see more.

Interesting Facts From the Oracle-BEA Strategy Roadmap Webcast

1. All the BEA dev community stuff, forums, etc. will be subsumed into Oracle's. That's sad, as Oracle's Web site is one of the worst implemented community sites on the Web. And I'm worried at Oracle's ability to truly support Fusion Middleware. Still at this late date, if you try to file a Sev1 SR with Oracle about your app server, it asks you "Is your database down?" as the criteria to accept it as Sev1. Middleware is a poor stepbrother on all of their online systems, perhaps the BEA user base will press them into doing better.

2. "Vast majority" of sales, support, and R&D staff from BEA are being retained.

3. Lots more info to come at OpenWorld in September and in some city events (see oracle.com/events/welcomeBEA - closest one to us is Dallas).

4. They say they're not going to just discontinue anything BEA, they'll support everything. Three levels to describe the disposition of each product - "Strategic Products" (use BEA and integrate into Fusion), "Continue & Converge" (rewrite it to integrate into Fusion) and "Maintenance" (just supporting it, mainly just stuff BEA had already EOLed).

5. Development Tools (25:00) - Their stated vision is a single complete & integrated dev environment, moving more toward "declarative" (aka visual, aka BPEL) development. So they're going with JDeveloper, ADF, and the new "Oracle Eclipse Pack" as the "Strategic" items. BEA Workshop's going into that. BEA was the second largest contributor to Eclipse. Thus by "complete and integrated" they mean "we'll continue to confusingly support both Eclipse and JDeveloper." Sigh.

6. App servers (29:30) - and a "transaction processing monitor" for C/C++/COBOL. (huh?) Oh, Tuxedo they mean. The strategic products are: BEA Tuxedo, BEA JRockit/realtime/liquid VM, WebLogic Server, Toplink, and Coherence. OAS/OC4J is listed as "Continue & Converge", which he explains as putting TopLink into WebLogic. So... OC4J is out, WebLogic is in! Woo! (My team members come and cluster excitedly around my computer at this revelation.) And they're going to go mainly JRockit for JVM, seems like - they say "We'll continue to support Sun's JVM... On Solaris."

They will continue to support OAS for those poor hapless ebusiness customers, there's no "forced migration" (but encouraged) there.

I cannot state how happy this is making me.

7. SOA! (39:00) "Now, it's all hot pluggable, you can mix and match, please don't hurt me."

Strategic products: Oracle Data Integrator, Oracle ESB ("unified" with AquaLogic ESB), Oracle BPEL, Complex Event Processor ("integrated" with WebLogic Event Server), and BAM. BEA WL-Integration is "C&C" and Cyclone is EOL. So "all Oracle baby."

Oh, actually, as he explains "unifying" the ESBs he means that pretty much putting AquaLogic on the Oracle SCA runtime and adding some of the Oracle functions, and the converged result will be a free upgrade for either customer. Nice.

And "integrated" event processor means really doing both, because WebLogic's is lighter/easier and Oracle's is heavier/more. (I'm unfamiliar with this space. Hmm.)

We use Oracle's SOA, so we use their ESB and BPEL products - I think we own the BAM but so far don't find it all that useful. For us the transition will be an ESB upgrade, which should be a lot less intrusive than an BPEL upgrade.

8. Business Process Management. Why is this listed different from SOA/BPEL? They get into depth on system-centric (e.g. order to cash), human-centric, document-centric (CMS), and decision-centric processes but say they want one product that will do them all. He describes some mutant hybrid of Oracle BPA designer, BEA BPM designer, "converged" Oracle/AquaLogic BPM, Oracle Document Capture & Imaging, Oracle Business Rules, Oracle Business Activity Monitoring (BAM), and WebCenter portal for viewing. This is going to be a mess, and I'll bet their "integration" here has 5 years to go before being worthy of the name.

9. Enterprise 2.0 and Portals. IMO Oracle "gets" Web 2.0 less than any of the big tech companies, so let's see what they have to say. They want one process to develop rich media, RIAs, enterprise portals, and social computing, with unified search and content management.

(You need to thank me for not having to listen to the labored ten minute "Web 2.0" scenario walkthrough where people use "forums" and "tags" and "RSS" other newfangled things...)

Strategic

• Oracle Universal Content Management for CM. A big "bah" from me here.
• Oracle WebCenter Framework. Portal, the least valuable part of any suite. Oracle's been trying to shove Portal down our throats for 5 years. We've given in and done POCs three times, and each time it's collapsed under its own weight.
• WebCenter Spaces & Suite. I'm sure their solution here is just lovely, since I've never heard of it.
• BEA Ensemble and Pathways - light-weight REST portal assembly. Hmmm, maybe should look at this. I see it's marketed as a mashup tool though, which isn't a good sign.

C&C

• BEA WL-Portal
• BEA AL-User Interaction

So in other words "we haven't been a player in Web 2.0 and aren't about to start." OK, fair enough.

10. Identity Management (1:08:00). We've been looking at this product suite; I'm not sure it's the best SSO solution out there but for someone with Oracle ERP it's mighty compelling. They've come a long way from the old DB-based OID/SSO crap they tried to sell a couple years ago.

Strategic products:

• OID (Oracle's LDAP)
• Oracle Identity Manager (account provisioning)
• Oracle Role Manager ("business role" provisioning)
• Oracle Access Manager (SSO)
• Oracle Adaptive Access Manager (risk-based strong authentication)
• BEA AL-Enterprise Security (authorization centralization)
• Oracle Identity Federation

Aka "Give us one meeeeeeellion dollars." That's the big problem with all this, you pretty much have to go all in on it and it needs a large team and lots of money to implement.

11. Systems Management (1:13:00). They even play in this space? Oh, they mean Enterprise Manager (sigh).

Strategic Products:

• EM Provisioning Pack (currently we just turn EM off on OAS, it's worthless)
• EM Configuration Pack, BEA Guardian integrated. We should look at this, currently doing builds, refreshes, etc. is a pain.
• EM Diagnostics Pack - adding App Diagnostics for Java and JRockit Mission Control. So far EM's Java app management is a joke, but perhaps the addition of these two (mainly MC) will help, they should be good.
• EM Management Pack for SOA (we looked at this and it's worthless)
• EM Management Pack for Identity
• EM Management Pack for BI

12. SOA Governance (1:18:00). Promoting reuse/portfolioing, operational control/policy. Ah, it's always better to buy expensive software than to think for five damn minutes about a process. But I'm not bitter.

• BEA AquaLogic Enterprise Repository (portfolio)
• Oracle Service Registry (UDDI), "partnership with HP" he says. ?
• OWSM (Web service security)
• EM Service Level Management Pack (bah)
• EM SOA Management Pack (worthless, see above)

13. Service Delivery Platform (1:23). Weird term for verticals, mainly telecom. I hesitate to even try to describe this stuff but for some reason you might use Oracle to deliver residential VOIP or PBX technology. If you're insane. They are uptaking the WebLogic SIP server, FYI.

Summary (1:28). They still are looking at Fusion Middleware as the overall suite and taking an integration approach. No change to the Fusion adoption strategy as a result, and some BEA bits are already certified by Oracle Applications. The new technologies should be 'transparent' to Apps customers, or are external addons. There is no forced migration to WebLogic for Oracle Apps, but it will be an option. In other words, don't get scared. (Not that we feel forced to upgrade to anything with our Oracle Apps install... The risk of upgrade keeps us on the oldest versions possible.)

Pricing is undergoing an upheaval, supposed to be "simplified" - in some cases it is, like single global pricing, but in some cases not, like going to their weird CPU/named user scheme.

They are rebranding and rereleasing the BEA products as 10gR3. and certifying everything under the BEA variant of 10gR3, and then the 11gR1/R2 will be more of a merge of the two branches.

And that's it!  Overall positive from my point of view.

Would you like to know more?

But Robert cued me in to PureText, which is a little Windows addon that strips all formatting from text when you cut/paste it for you.  By default you Windows+V instead of Control+V and voila, no crap.  Yay!

Several months ago we evaluated a bunch of log management solutions with several goals in mind. We wanted a solution that was agile enough to be able to take in a wide variety of log formats as well as configuration files. It needed to shield sensitive information (passwords, credit card information, etc) from unauthorized users. It needed to provide us with a customizable interface where we could report on all of the log data it gathered. Lastly, it needed to provide our customers (developers) with the ability to self-service their own log files. After evaluating most of the major players in the log management arena, we found our ideal solution in a product called Splunk.

The first thing I noticed when evaluating Splunk was that they're not like everyone else. They're not trying to sell you some sort of logging appliance and they offer their software free for customers with 100 MB/day or less worth of logging. Getting Splunk installed was a breeze. You can have it up and running in minutes. It truly is Log Management for Dummies in that respect, but under the hood there is a highly configurable and customizable tool with an API that you could use to write your own applications to examine log files.

At this point I've mucked around with Splunk for a few months and our configuration is pretty intense. I've added in custom indexes to make my custom dashboards load faster. I've set Splunk up to create queryable metadata fields based on information in the logs. I've added filters for custom timestamps and auditing so we can tell if a log file has been modified. I've even set up a "deployment server" to distribute Splunk's configuration bundles to my various types of servers. This brings me to the one drawback of Splunk: Upgrading. Rumor has it that they are working on making it easier to upgrade from one version to the next, but for the time being it involves logging in to each server, stopping Splunk, upgrading the files, and restarting Splunk again. If you only had to upgrade every once in a while it would be fine, but they maintain a very active development team so I find myself constantly wanting to upgrade to get the latest bug fixes and features.

Other than that, Splunk does exactly what I tell it to do. It grabs all of our logs and presents them in a single intuitive interface. Think of it as a search engine for log and configuration files. Then, once I have the log data in front of me, I can create custom reports based on that data. If I want to, I can even alert based on information Splunk finds in my logs (send an e-mail to a developer every time their application throws an error message). Oh, did I mention that Splunk has a PCI Dashboard that you can install for free? Ask those other guys how much they charge for their PCI solution.

The next time you have some free time be sure to download Splunk and install it on one of your development servers. You won't be disappointed.