We've been working on a deal with one specific supplier.  As part of it, they'll be hosting some images for our site.  There's a business team primarily responsible for evaluating their functionality etc., we're just in the mix as the faithful watchdogs of performance and availability for our site.

Round 1 - "What are these SLAs you speak of?"  The vendor offers no SLA.  "Unacceptable," we tell the project team.  They fret about having to worry about that along with the 100 other details of coming to an agreement with the supplier, but duly go back and squeeze them.  It takes a couple squeezes because the supplier likes to forget about this topic - send a list of five questions with one of them being "SLA," you get four answers back, ignoring the SLA question.

Round 2 - "Oh, you said 'SLA'!  Oh, sure, we have one of those."  We read the SLA and it only commits to their main host being pingable.  Our service could be completely down, and it doesn't speak to that.  Back to our project team, who now between the business users, procurement agent, and legal guy need more urging to lean on the supplier.  The supplier plays dumb for a while, and then...

Round 3 - "Oh, performance and availability of the service we're supposed to be providing you!  Yeah, we have that."  From somewhere comes a huge set of legalese with definitions of "to the glass" performance and everything.  Until this week they had "no idea" what we meant about a service performance SLA.  So we read that - the definitions look good, but now we go down to the remedies.  They define all these performance metrics, but down in the clause that says "you get money back if we jack it up" they carefully only list their total ping outages.  And you can only get compensation for one of these total outages if you report it *during* the outage.  And if you do that, you get a credit for 1/10 of your monthly bill.  No dice.  So back to the project team.  The procurement agent is very concerned that having to continuously work on this will interefere with has carefully crafted deal.  "Sorry, no SLA no go," we say.  Meetings worth of internal friction occur, until we go back to the vendor yet again.

Here's where you get into the truly deceptive territory.  If you read the SLA, up front you see all these definitions and tables about "to the glass performance" and "over DSL!" and everything so you think "great, it's taken into account!"  But their lawyers have done their job well, so they can put in all the stuff they want but if down in the bottom it doesn't say "and if we don't live up to that we give you money back" it's worthless.

Round 4 - Still pending.  But we've seen this all before, this supplier isn't unique by any means.  We'll get another two drafts in before we're done, assuming that our business users don't freak out and just say "we'll accept the risk!!!" and sign the contract.

Next will come up how it's measured.  The supplier will say "we'll measure it!  Trust us!"  Obviously that's stupid.  We usually pay for a Keynote or similar monitor as an impartial third party (expensive, but less expensive than a sucky SaaS service).  Then they'll try one final draft where they'll say they're accepting all our terms but will cleverly revert one of the earlier edits to break the link between definitions and remedies.  It's like there's some script they all follow.

And every supplier does this.  It's how they "protect" themselves.  This isn't a fly by night operation, it's a large supplier and 90% of you have their software loaded up on your PC right now.   They rely on you either not bothering with the SLA in the first place, or not reading it carefully enough, or not having the gumption to go 6 rounds with them to get an enforceable one in place.  That's good odds for them.  Don't accept it.  You're paying for a service and you deserve to get that service, and get your money back if they don't supply it in a usable way.

Here's some things to keep in mind when selecting a SaaS vendor, laced with real life horror stories from our experiences.

1.  Performance/Availability.  Set a hard performance/availability SLA in the contract.  Many vendors won't even have an SLA clause, or they'll have one that says "99.9% uptime!" without any remedy clause for what if they don't hit that.  You want a clear SLA with a clear measurement method and clear "money back" if they don't hit it.  We use a 2 second global performance SLA as measured by a Keynote Global 35 monitor.  But the SLA isn't the whole story - you are counting on these people to accomplish your goals.

True story.  We did an implementation with a new SaaS supplier.  Everything looked fine but the team had not done performance testing.  With only a week to go live, they finally loaded in a full set of data into the system and saw performance that was horrible, clearly in the ~30 second page load time even in the US (much worse internationally).  But it isn't as simple as "Oh, you're not hitting the SLA, goodbye..."  We have months of time invested in this supplier.

2.  Privacy policy.  Make sure you know what they're going to do with your user data they collect and make sure that matches what your privacy policy says.  We're international and so have to abide by a fairly restrictive EU-compliant privacy policy.  Many SaaS vendors don't know what exactly that entails, and so can run you afoul - remember, you're legally responsible for your site whether you've outsourced parts of it or not.

3.  Security.  One, if you have compliance concerns like PCI etc. you need to make sure they're complaint as well, and certified as such by an auditor (don't take "Oh, yeah, we're PCI complaint" at anyone's word).  Two, you are almost certainly going to have to exchange user credentials, so a user can log in across you site and the SaaS site with the same login, and the "right ways" to do this, like SAML, are supported by about one tenth of one percent of SaaS vendors.  You need to carefully review how you're going to do it to avoid security problems.

4.  Contingency plan.  Companies go out of business, or the relationship between them degrades.  You have to have a plan in place for when your SaaS vendor dies, gets bought by HP and has their price quadrupled,or you decide you hate each other, or the cost isn't what you anticipated (see point 5 below).   You need some of this in your contract - in any event, you get your data andif they are dying, a perpetual license to their software.  (If you're really lucky it's one of these firms that has a software package and an ASP version both.)

True story.  Our ni.com forums were hosted for many years by a company called QUIQ.  In the big bubble bust, they went down and had the revenooers show up to unplug them.   We were very lucky in that they were interested in helping us despite their own problems, and that we had bad ass technical folks on staff.  We paid to have a QUIQ engineer come down and load up their forum software on our systems (and this wasn't a bundled software solution, it was their internal-only stuff) and transition our data over.  We then supported this as our forum solution for a couple years, often having to take measures like decompiling the Java to make changes.   But that was better than being dead in the water.  Now with our new forums vendor, we do things like get regular backups of our data, gateway the forum to NNTP, etc.

5.  Cost.  SaaS vendors almost universally charge you by usage.  Which is "fair" - but the Internet is a wild place.  What about when that new Chinese experimental spider (soso, you suck!) decides to grant you a couple hundred thousand extra page views one day?  You get stuck with the bill.  Corporations have to be able to budget for their expenditures, and there is risk in thsi business model that one time events and/or unexpected growth will fundamentally alter your cost and ROI structure.

There are several potential mitigations here.  One is to get a SaaS vendor that implements throttling, so you can meter back untoward amounts of traffic.  Another is to have specific payment scales that mitigate this- you want to avoid "cell phone" type plans that give you a certain amount and then an abusive overuse charge like the plague.  Look for plans like ISPs, where you pay for the 90th percentile peak of traffic, for example.  Or have a prepay agreement, and/or specify what kinds of traffic you'll pay for.  (On many Internet sites, spiders account for 30% or so of the traffic and thus the expense.)

6.  Quality.  Do a pilot/proof of concept.  DO ONE!!!  A SaaS vendor is not yet a commodity in most areas.  You are getting as deep in bed with them as if you bought hardware and software and in house programming.  Don't sign the contract until you have seen it work for you.  Build deliverables and payment schedule into the contract - "You get 1/3 upon requirement completion, 1/3 upon signoff of a test version, and 1/3 upon go live" is popular.  We have one purchasing agent who likes to push "Sign the contract, and have a 30 day 'out' clause if things don't go well."  But this ignores the deep investment into even a SaaS vendor (and the fact that most implementations aren't fully baked in 30 days).  As a result, we have several business units even now signing up with ASPs without doing a formal pilot, and every one of them will come to regret it bitterly.  Only the rich and the idiotic buy a car without a test drive and a mechanic checkup.  Any SaaS solution will be much more expensive than any car.

None of this is to say that SaaS is bad or should be avoided.  But it needs to be evaluated just like any other solution.  Is the 5 year TCO really better than in house, not just the first year cost?  And does it really do what you need - functionality wise, but also in the areas of performance, security, and these other areas which make your functionality meaningful to the users?  If you can't answer these questions, you are betting a lot of money and your reputation on an untried horse.  Find out the answers.