One of the reasons that our FAST enterprise search program has been so successful here is that the programmers and the Web Admins have worked pretty much 50/50 on the platform.  Also, FAST is a great product and has great support (we're waiting with bated breath to see if Microsoft screws it up; we've been with FAST since way way before they got bought), and we have some very visionary search business folks who saw its potential early on.

Nowadays, search is more than it was considered traditionally.  We have a normal "search box", of course.   But we also run our faceted navigation off search (e.g. our Data Acquisition product line page), pull things like related links and other resources (see resources tab on this page).  Search, in many ways, can be used the way people have used databases in the past.  With some metadata added, a search index is kinda like a big database, highly denormalized for speed, focusing on text search.  In fact, I think there's a master's thesis in there somewhere as to when search makes sense vs. when a database makes sense.  Databases make sense with lots of numerical information, but on the Web that's frankly a fringe use case!   On the Web it's all about text, from name/address to links to articles to product info...  When we did things like query related links out of a database table, and I mean an oracle database table on a big ass Solaris box, it was painfully slow.  Pulling from search, it's 15 milliseconds.

As a result, our internal search use is even more killer.  We pull Intranet pages, documents from Notes repositories, data from our Oracle ERP system, files off file shares, etc. all into one place and let people delve through it.  They've even implemented "screens" on top of some of the data (mainly because Oracle ERP is painful to use).  Our entire sales force is gaga over it.

Anyway, so yay to modern search technology, yay to FAST, and yay us!

Let me begin by explaining the why before I tell you the how.  In order for a search engine to index a site's content, it needs to be able to see that content.  The webmasters of that site are eager to let the search engine see the content as they know it will drive additional visitors to their site.  The end result is that they have to find a way for the search engine to see the content, while at the same time obscuring it from the view of the average user.  Most of the time they do this by keying off of the browser's USER AGENT.  This creates a loophole for us to exploit since if Google is able to see the search engine results, then so can we.  Here's my two tricks to see the restricted content:

• Trick #1: Change the user agent! In Firefox, this can be done rather easily by entering "about:config" as an address in the address bar.  Click the right mouse button to get the context menu and then select New->String.  Enter the preference name "general.useragent.override".  Now simply enter the name of the new user agent string that you want to use.  If you want to cloak yourself as Google and see what they see, try using the string "Googlebot/2.1 (+http://www.googlebot.com/bot.html)".  You can check the current value by just entering "about:" in the address bar.
• Trick #2: Check Google's cache! If you don't want to go about mucking with your browser's preferences, there's an even easier trick for you to use.  Go to http://www.google.com and enter the restricted URL, but prepend "cache:" before it.  This allows you to pull the site's indexed content out of Google's cache instead of from the site itself, bypassing the access restrictions entirely.

So, there you have it.  Two simple ways to read restricted website content without ever having to register an account with the site.  Enjoy!