Oracle + BEA Update
A year ago I wrote about Oracle's plan on how to combine BEA Weblogic and OAS. A long time went by before any more information appeared - we met with our Oracle reps last week to figure out what the deal is. The answer wasn't much more clear than it was way back last year. They do certainly want some kind of money to "upgrade" but it seems poorly thought through.
OAS came in various versions - Java, Standard, Standard One, Enterprise, and then the SOA Suite versions. The new BEA, now "Fusion Middleware 11g" comes in different versions as well.
- WLS Standard
- WLS Enterprise - adds clustering, costs double
- WLS Suite - adds Coherence, Enterprise Manager, and JRockit realtime, costs quadruple
But they can't tell us what OAS product maps to what FMW version.
There is also an oddly stripped down "Basic" edition which noted as being a free upgrade from OAS SE but it strips out a lot of JMS and WS stuff; there's an entire slide of stuff that gets stripped out and it's hard to say if this would be feasible for us.
As for SOA Suite, "We totally just don't know."
Come on Oracle, you've had a year to get this put together. It's pretty simple, there's not all that many older and newer products. I suspect they're being vague so they can feel out how much $$ they can get out of people for the upgrade. Hate to break it to you guys - the answer is $0. We didn't pay for OAS upgrades before this, we just paid you the generous 22% a year maintenance that got you your 51% profit margin this year. If you're retiring OAS for BEA in all but name, we expect to get the equivalent functionality for our continued 22%.
Oracle has two (well, three) clear to dos.
1. Figure out what BEA product bundles give functionality equivalent to old OAS bundles
2. Give those to support-paying customers
3. Profit. You're making plenty without trying to upcharge customers. Don't try it.
Customizing Apache Error Codes By URL
I've had a couple of discussions lately about customized Apache error pages that prompted me to do a little bit of research on it. What I've come up with is somewhat interesting so I thought I'd share it with everyone. First, it is not technically possible to tell Apache to serve up a different error page for image content than for html content than for php content since the only command Apache accepts for this is of the "ErrorDocument error-code document" format. That said, if you allow .htaccess overrides on a particular directory, then you can specify your ErrorDocument directive in there as well; overriding the default error handling specified in the httpd.conf file. An example:
In my httpd.conf file I have all 404's going to errorpage.cgi with the following line:
ErrorDocument 404 /cgi-bin/errorpage.cgi
I'm a good little code monkey and put all of my images in a /images directory under the DirectoryRoot. By default, if I were to hit a non-existent image in that directory, I would get the default error message defined in the httpd.conf file. If that image were referenced in an html page that I hit, I now download the html page plus the errorpage.cgi page for the bad image reference, introducing one whole page's worth of additional overhead.
But since I was a good code monkey and put all of my images in a /images directory, the fix for this is really simple. I create a .htaccess file inside of my /images directory and add the following line to it:
ErrorDocument 404 "404 - Image does not exist <-- Note: No end quote is intentional
Now, if I hit http://www.mysite.com/badpage.html I get the errorpage.cgi page, but if I hit http://www.mysite.com/images/badimage.jpg I get a short and sweet message saying "404 - Image does not exist". I haven't tested this yet to see how it works when you are using something like mod_oc4j to send certain URLs to an application server, but it's possible that this could work there too if Apache checks for existing static URLs before passing requests to the app server. Further testing could be useful there.
So there you have it. I can't tell Apache to serve up different error pages based on the URL or file type, but if I'm diligent about putting different files under different directories, I can effectively do the same thing using .htaccess files. Woot!
Google Chrome Hates You (Error 320)
The 1.0 release of Google Chrome has everyone abuzz. Here at NI, loads of people are adopting it. Shortly after it went gold, we started to hear from users that they were having problems with our internal collaboration solution, based on the Atlassian Confluence wiki product. They'd hit a page and get a terse error, which if you clicked on "More Details" you got the slightly more helpful, or at least Googleable, string "Error 320 (net::ERR_INVALID_RESPONSE): Unknown error."
At first, it seemed like if people reloaded or cleared cache the problem went away. It turned out this wasn't true - we have two load balanced servers in a cluster serving this site. One server worked in Chrome and the other didn't; reloading or otherwise breaking persistence just got you the working server for a time. But both servers worked perfectly in IE and Firefox (every version we have lying around).
So we started researching. Both servers were as identical as we could make them. Was it a Confluence bug? No, we have phpBB on both servers and it showed the same behavior - so it looked like an Apache level problem.
Sure enough, I looked in the logs. The error didn't generate an Apache error, it was still considered a 200 OK response, but when I compared the log strings the box that Chrome was erroring on showed that the cookie wasn't being passed up; that field was blank (it was populated with the cookie value on the other box and on both boxes when hit in IE/Firefox). Both boxes had an identically compiled Apache 2.0.61. I diffed all the config files- except for boxname and IP, no difference. The problem persisted for more than a week.
We did a graceful Apache restart for kicks - no effect. Desperate, we did a full Apache stop/start - and the problem disappeared! Not sure for how long. If it recurs, I'll take a packet trace and see if Chrome is just not sending the cookie, or sending it partially, or sending it and it's Apache jacking up... But it's strange there would be an Apache-end problem that only Chrome would experience.
I see a number of posts out there in the wide world about this issue; people have seen this Chrome behavior in YouTube, Lycos, etc. Mostly they think that reloading/clearing cache fixes it but I suspect that those services also have large load balanced clusters, and by luck of the draw they're just getting a "good" one.
Any other server admins out there having Chrome issues, and can confirm this? I'd be real interested in knowing what Web servers/versions it's affecting. And a packet trace of a "bad" hit would probably show the root cause. I suspect for some reason Chrome is partially sending the cookie or whatnot, choking the hit.
Using Proxies to Secure Applications and More
I've been really surprised that for as long as I've been active with OWASP, I've never seen a proxy presentation. After all, they are hugely beneficial in doing web application penetration testing and they're really not that difficult to use. Take TamperData for example. It's just a firefox plugin, but it does header, cookie, get, and post manipulation just as well as WebScarab. Or Google Ratproxy, which works in the background while you browse around QA'ing your web site and gives you a nice actionable report when you're done. I decided it was time to educate my peers on the awesomeness of proxies.
This past Tuesday I presented to a crowd of about 35 people at the Austin OWASP Meeting. The title of my presentation was "Using Proxies to Secure Applications and More". Since so many people came up to me afterward telling me what a great presentation it was and how they learned something they can take back to the office, I decided (with a little insistance from Ernest) that it was worth putting up on SlideShare and posting to the Web Admin Blog.
The presentation starts off with a brief description of what a proxy is. Then, I talked about the different types of proxies. Then, the bulk of the presentation was just me giving examples and demonstrating the various proxies. I included anonymizing proxies, reverse proxies, and intercepting proxies. While my slides can't substitue for the actual demo, I did try to include in them what tool I used for the demo. If you have any specific questions, please let me know. All that said, here's the presentation.
Google Ratproxy
If you are responsible for developing or maintaining a website and haven't checked out Ratproxy yet, you're missing out. Before I start spouting off about just how cool and useful this tool is, I suppose I should first tell you what a proxy is. In a nutshell, a proxy is an application that runs local on your computer and intercepts requests and responses between your web browser and the web server. In almost all cases, the proxy has the ability to manipulate the conversation going on between the two. Things like modifying your cookies, changing POST and GET parameters, and finding hidden fields are made uber-easy with the assistance of a proxy.
Oracle + BEA = ?
We use Oracle Application Server as our Java app server at NI. Yeah, yeah, I'll wait till you stop laughing.
Why not JBoss or WebLogic or WebSphere? Well, a couple reasons. We made the decision five years ago, and JBoss wasn't solid then, and we needed J2EE support so plain Tomcat wasn't enough. And we're a huge Oracle shop and figured that if we were using the same app server on the Web and our ERP tiers there'd be leverage in terms of developer knowledge etc. Would we make that same decision today? I'm not sure about that (I can hear my team members shouting "hell no" over the cube walls). Although since we've also gone with Oracle's SOA suite for ESB and BPEL it would be harder to switch. But still tempting - Oracle has done a horrible job in getting their app server supported by other vendors. Every time we buy something and look at the supported app server section of their support matrix, and we ask "What about Oracle's OAS?" we get expressions of mixed horror and pity from the supplier. (I liked it when the Chinese technical guy from one eComm vendor we had in responded to this question with, "You know, the Tomcat is good, and free! Maybe you use that!")
Anyway, Oracle bought BEA a while back, which got keen interest from us. Stay with Oracle *and* use a good app server that other people support? Tempting! But Oracle's been farting around for six months without coming out with a statement on what this will mean for the products. Oracle's finally done a Webcast describing their strategy. Well, it's half marketing and a celebration of how many million dollars they have. But there's also a lot of product strategy in there. I'll sum it up for you because the damn webcast is nearly two hours long, and I don't want other people to have to waste that much time on it. Unless you like to hear someone go on about "strategic clarity" and "customer profiles," in which case this is two hours of bliss for you and you should watch it. Although I also had the stream break a bunch of times while watching. Who the heck uses RealPlayer any more? Anyway, here's a list of the interesting product facts from the Webcast. Some are marked with their timestamp if you want to fast forward to them and see more.
Quick Blogging Tip
All yesterday I was being annoyed by the need to write up my blog posts in another editor and paste them over into Wordpress. You have to do that because composing text longer than about 3 sentences in a browser window is taking your life in your hands. But I discovered even in cutting and pasting from Wordpad you get bullshit formatting inserted that drives the TinyMCE editor crazy. And Notepad was giving me line break problems. (And it needs not be said that you should never ever paste from Word...)
But Robert cued me in to PureText, which is a little Windows addon that strips all formatting from text when you cut/paste it for you. By default you Windows+V instead of Control+V and voila, no crap. Yay!
Log Management for Dummies (aka Splunk)
Logs are one thing that I think are severely underutilized by most systems administrators. Most of us have taken the first step by actually logging the data, but neglect organizing it into any sort of manageable form. You'll probably argue that any hardcore *nix admin would be able to take the raw logs using grep, cut, awk, and a handful of other *nix power tools and turn it into consumable information, but that'll only get you so far.
Several months ago we evaluated a bunch of log management solutions with several goals in mind. We wanted a solution that was agile enough to be able to take in a wide variety of log formats as well as configuration files. It needed to shield sensitive information (passwords, credit card information, etc) from unauthorized users. It needed to provide us with a customizable interface where we could report on all of the log data it gathered. Lastly, it needed to provide our customers (developers) with the ability to self-service their own log files. After evaluating most of the major players in the log management arena, we found our ideal solution in a product called Splunk.
The first thing I noticed when evaluating Splunk was that they're not like everyone else. They're not trying to sell you some sort of logging appliance and they offer their software free for customers with 100 MB/day or less worth of logging. Getting Splunk installed was a breeze. You can have it up and running in minutes. It truly is Log Management for Dummies in that respect, but under the hood there is a highly configurable and customizable tool with an API that you could use to write your own applications to examine log files.
At this point I've mucked around with Splunk for a few months and our configuration is pretty intense. I've added in custom indexes to make my custom dashboards load faster. I've set Splunk up to create queryable metadata fields based on information in the logs. I've added filters for custom timestamps and auditing so we can tell if a log file has been modified. I've even set up a "deployment server" to distribute Splunk's configuration bundles to my various types of servers. This brings me to the one drawback of Splunk: Upgrading. Rumor has it that they are working on making it easier to upgrade from one version to the next, but for the time being it involves logging in to each server, stopping Splunk, upgrading the files, and restarting Splunk again. If you only had to upgrade every once in a while it would be fine, but they maintain a very active development team so I find myself constantly wanting to upgrade to get the latest bug fixes and features.
Other than that, Splunk does exactly what I tell it to do. It grabs all of our logs and presents them in a single intuitive interface. Think of it as a search engine for log and configuration files. Then, once I have the log data in front of me, I can create custom reports based on that data. If I want to, I can even alert based on information Splunk finds in my logs (send an e-mail to a developer every time their application throws an error message). Oh, did I mention that Splunk has a PCI Dashboard that you can install for free? Ask those other guys how much they charge for their PCI solution.
The next time you have some free time be sure to download Splunk and install it on one of your development servers. You won't be disappointed.