They've been doing some interim freebie Web conferences and there's one coming up.  Check it out.  They'll be talking about performance functionality in Google Webmaster Tools, mySQL, Show Slow, provisioning tools, and dynaTrace's new AJAX performance analysis tool.

O'Reilly Velocity Online Conference: "Speed and Stability"
Thursday, March 17; 9:00am PST
Cost: Free

• Online apps are complex systems
• A siloed approach of deciding to improve midtier vs CDN vs front end engineering results in suboptimal experience to the end user - have to take holistic view.  I totally agree with this, in our own caching project we took special care to do an analysis project first where we evaluated impact and benefit of each of these items not only in isolation but together so we'd know where we should expend effort.
• Use top level/end user metrics, not system metrics, to measure performance.
• There are other metrics that correlate to your performance - "key indicators."
• It's hard to take low level metrics and take them "up" into a meaningful picture of user experience.

He's covering good stuff but it's nothing I don't know.  We see the differences and benefits in point in time tools, Passive RUM, tagging RUM, synthetic monitoring, end user/last mile synthetic monitoring...  If you don't, read the presentation, it's good.  As for me, it's off to the scaling session.

I hopped into this session a half hour late.  It's Scalable Internet Architectures (again, go get the presentation) by Theo Schlossnagle, CEO of OmniTI and author of the similarly named book.

I like his talk, it starts by getting to the heart of what Web Operations - what we call "Web Admin" hereabouts - is.  It kinda confuses architecture and operations initially but maybe that's because I came in late.

He talks about knowledge, tools, experience, and discipline, and mentions that discipline is the most lacking element in the field. Like him, I'm a "real engineer" who went into IT so I agree vigorously.

What specifically should you do?

• Use version control
• Monitor
• Serve static content using a CDN, and behind that a reverse proxy and behind that peer based HA.  Distribute DNS for global distribution.
• Dynamic content - now it's time for optimization.

Optimizing Dynamic Content

Don't pay to generate the same content twice - use caching.  Generate content only when things change and break the system into components so you can cache appropriately.

example: a php news site - articles are in oracle, personalization on each page, top new forum posts in a sidebar.

Why abuse oracle by hitting it every page view?  updates are controlled.  The page should pull user prefs from a cookie.  (p.s. rewrite your query strings)
But it's still slow to pull from the db vs hardcoding it.
All blog sw does this, for example
Check for a hardcoded php page - if it's not there, run something that puts it there.  Still dynamically puts in user personalization from the cookie.  In the preso he provides details on how to do this.
Do cache invalidation on content change, use a message queuing system like openAMQ for async writes.
Apache is now the bottleneck - use APC (alternative php cache)
or use memcached - he says no timeouts!  Or... be careful about them!  Or something.

Scaling Databases

1. shard them
2. shoot yourself

Sharding, or breaking your data up by range across many databases, means you throw away relational constraints and that's sad.  Get over it.

You may not need relations - use files fool!  Or other options like couchdb, etc.  Or hadoop, from the previous workshop!

Vertically scale first by:

• not hitting the damn db!
• run a good db.  postgres!  not mySQL boo-yah!

When you have to go horizontal, partition right - more than one shard shouldn't answer an oltp question.   If that's not possible, consider duplication.

IM example.  Store messages sharded by recipient.  But then the sender wants to see them too and that's an expensive operation - so just store them twice!!!

But if it's not that simple, partitioning can hose you.

Do math and simulate it before you do it fool!   Be an engineer!

Multi-master replication doesn't work right.  But it's getting closer.

Networking

The network's part of it, can't forget it.

Of course if you're using Ruby on Rails the network will never make your app suck more.  Heh, the random drive-by disses rile the crowd up.

A single machine can push a gig.  More isn't hard with aggregated ports.  Apache too, serving static files.  Load balancers too.  How to get to 10 or 20 Gbps though?  All the drivers and firmware suck.  Buy an expensive LB?

Use routing.  It supports naive LB'ing.  Or routing protocol on front end cache/LBs talking to your edge router.  Use hashed routes upstream.  User caches use same IP.  Fault tolerant, distributed load, free.

Use isolation for floods.  Set up a surge net.  Route out based on MAC.  Used vs DDoSes.

Service Decoupling

One of the most overlooked techniques for scalable systems.  Why do now what you can postpone till later?

Break transaction into parts.  Queue info.  Process queues behind the scenes.  Messaging!  There's different options - AMQP, Spread, JMS.  Specifically good message queuing options are:

• ActiveMQ (Java)
• OpenAMQ (C)
• RabbitMQ (erlang)

Most common - STOMP, sucks but universal.

Combine a queue and a job dispatcher to make this happen.  Side note - Gearman, while cool, doesn't do this - it dispatches work but it doesn't decouple action from outcome - should be used to scale work that can't be decoupled.  (Yes it does, says dude in crowd.)

Scalability Problems

It often boils down to "don't be an idiot."  His words not mine.  I like this guy. Performance is easier than scaling.  Extremely high perf systems tend to be easier to scale because they don't have to scale as much.

e.g. An email marketing campaign with an URL not ending in a trailing slash.  Guess what, you just doubled your hits.  Use the damn trailing slash to avoid 302s.

How do you stop everyone from being an idiot though?  Every person who sends a mass email from your company?  That's our problem  - with more than fifty programmers and business people generating apps and content for our Web site, there is always a weakest link.

Caching should be controlled not prevented in nearly any circumstance.

Understand the problem.  going from 100k to 10MM users - don't just bucketize in small chunks and assume it will scale.  Allow for margin for error.  Designing for 100x or 1000x requires a profound understanding of the problem.

Example - I plan for a traffic spike of 3000 new visitors/sec.  My page is about 300k.  CPU bound.  8ms service time.  Calculate servers needed.  If I varnish the static assets, the calculation says I need 3-4 machines.  But do the math and it's 8 GB/sec of throughput.  No way.  At 1.5MM packets/sec - the firewall dies.  You have to keep the whole system in mind.

So spread out static resources across multiple datacenters, agg'd pipes.
The rest is only 350 Mbps, 75k packets per second, doable - except the 302 adds 50% overage in packets per sec.

Last bonus thought - use zfs/dtrace for dbs, so run them on solaris!

Peco, my erstwhile Bulgarian comrade, and I came in to San Jose  from Austin on Sunday.  We got situated at the fairly swank hotel, the Fairmont, and wandered out to find food.  There was some festival going on so the area was really hopping.  After a bit of wandering, we had a reasonably tasty dinner at Original Joe's.  Then we walked around the cool pedestrian part of downtown San Jose and ended up watching "Terminator:  Salvation" at a neighborhood movie theater.

We went down at 8  AM the next morning for registration.  We saw good ol' Steve Souders, and hooked up with a big crew from BazaarVoice, a local Austin startup that's doing well.  (P.S. I don't know who that hot brunette is in the lead image on their home page, but I can clearly tell that she wants me!)

This first day is an optional "workshop" day with a number of in depth 90 minute sessions.  There were two tracks, operations and performance.   Mostly I covered ops and Peco covered performance.  Next time - the first session!

As Web Admins, we love Velocity.  Usually, we have to bottom-feed at more generalized conferences looking for good relevant content on systems engineering.  This is the only conference that is targeted right at us, and has a dual focus of performance and operations.  The economy's hitting us hard this year and we could only do one conference - so this is the one we picked.

Look for full coverage on the sessions to come!

Our max inbound Internet throughput was 285 Mbps, about double our usual peak.  We saw a ni.com Web site performance degradation of about 25% for less than two hours according to our Keynote stats.  ni.com ASPs were affected proportionately which indicates the slowdown was Internet-wide and not unique to our specific Internet connection here in Austin.  The slowdown was less pronounced internationally, but still visible.  So in summary - not a global holocaust, but a noticeable bump.

Cacti graphs showing our Internet connection traffic:

Keynote graph of several of our Web assets, showing global response time in seconds:Looking at the traffic specifically, there were two main standouts.  We had TCP 1935, which is Flash RTMP, peaking around 85 Mbps, and UDP 8247, which is a special CNN port (they use a plugin called "Octoshape" with their Flash streaming), peaking at 50 Mbps.   We have an overall presence of about 2500 people here at our Austin HQ on an average day, but we can't tell exactly how many were streaming.  (Our NetQoS setup shows us there were 13,600 'flows,' but every time a stream stops and starts that creates a new one - and the streams were hiccupping like crazy.  We'd have to do a bunch of Excel work to figure out max concurrent, and have better things to do.)

In terms of the streaming provider breakdown - since everyone uses Akamai now, the vast majority showed as "Akamai".  We could probably dig more to find out, but we don't really care all that much.  And, many of the sources were overwhelmed, which helped some.

We just wanted to share the data, in case anyone finds it helpful or interesting.

The conference was quite a success. Everyone we spoke to was enthusiastic about the people and information there. O'Reilly is happy because attendance was above their expectations, and it looks like it's been expanded to 3 days next year, which is good - it was *so* session packed and fast paced I didn't get to talk to all the suppliers I wanted in the dealer room and at times it felt like the Bataan death march. The first day we barely had time to grab a fast food dinner, and we often found ourselves hungry and hurrying. We enjoyed talking with the people there, but it seemed less conversational than other conventions - maybe because of the pace, maybe because half the people there were from the area and thus needed to scamper off to work/home and were therefore not into small talk.

There were the usual forgivable rough spots of a first year conference - programming varied from 15 to 45 minute slots and some of the sessions were shoehorned into a too-small spot or something meriting more of an Ignite format given a long slot. Really, the only thing that bothered me was how much one of the two rooms (both carved out of the same superspace in traditional hotel style) sucked compared to the other. The main room was great - big, good AC, good sound. The second room was a nightmare - very long, about 85 degrees, and poor sound. I had to go out and complain once to the staff about at least getting the AC going in there. The Marriott was very nice overall but that one room kept me away unless the subject was really compelling. And sometimes, a talk most people were fleeing from (SOX, I'm looking at you) was in the big room while the session that everyone was in was in the small one.

What were my biggest takeaways. Well, as always, a bunch of opinions on what tools are good to be looking into. Jiffy sounds neat, we're already using KITE, and we're evaluating proxy caches and CDNs so all the info on those was great. It was also a good opportunity to touch base with suppliers old and new - it really helps the relationship to be able to interact with them out "in the wild," not over the phone or on a sales call. We talked at length with Keynote and Coradiant, current supplier partners of ours; got to talk with prospectives like Akamai and LimeLight, and got t meet new ones we were impressed with like ControlTier. And the EFF even had a booth. Fight the power!

A lot of the talks were too short to really get useful specifics out of per se, but they serve as valuable pointers to further information. The image optimization talk by Yahoo! was one of those, we're going to be looking into that stuff hard. And Steve's Even Faster talk. And all the random name-dropped tools, open source and not, that people are getting mileage out of.

You know, a side note to that - it sucks that there's not a semi-definitive list of a "Web Admin's Toolkit." What areas you need to have solutions in (monitoring, configuration management, proxies, etc.). Maybe we'll try to put together something like that, a multidimensional vote-driven thing where people can rate their faves all in one place.

In closing, I'd like to thank O'Reilly for putting together a conference that serves a growing and largely unserved part of the technical populace, and specifically to Steve for spearheading it and asking me to speak!

We hear about Capacity Planning with John Allspaw of Flickr. He says: No benchmarks! Use real production data. (How? We had to develop a program called WebReplay to do this because no one had anything. We're open sourcing it soon, stay tuned.)

Use "safety factors" (from traditional engineering). Aka a reserve, overhead, etc.

They use squid a bunch. At NI we've been looking at Oracle's WebCache - mainly because it supports ESIs and we're thinking that may be a good way to go. There's a half assed ESI plugin for squid but we hear it doesn't work; apparently Zope paid for ESI support in squid 3.0 but no traction on that in 4 years best as we can tell. But I'd be happy not to spend the money.

Anyway. You should do forecasting. Assuming it's linear which it never is. But you can take output from your stuff (ganglia, whatever) and fityk can give you a curve fit.

They use lots of nagios for monitoring - about 10 checks per host.

Determine a ceiling, and high/low water marks, and alert outside the water marks.

Then have a simple capacity dashboard for everything - how close to ceiling you are.

Horizontal scaling is all well and good, but sometimes you should do some vertical by upgrading. He calls it "diagonal." By upgrading image proc servers, they got same CPU usage but 3x more work out of them. (We saw the same when we upgraded our Java app servers from Sun V440s to Dell 2850s a year or two ago - 50% performance improvement.) In their case, they also got faster processing time, less power usage, and less rack space.

Memcached. You turn it on, and the DBs go idle! Yay. But then your Web servers heat up as they become the bottleneck. So beware the wandering bottleneck.

Stupid Capacity Tricks! Before Puppet and Capistrano there was dsh (distributed shell). Ooo, I want it.

Shut Shit Off - they have software switches to disable various features when needed. (We have a lot of those switches at NI, but they're not documented and under the control of business units not ops - sad.) Their programmers are good, they put flags in config files in order of importance to turn things on/off, read on the fly.

Host an outage page NOT in your datacenter, and use it - users appreciate knowing what's up.

Bake dynamic into static. Some Yahoo! properties have a big red button to bake/unbake at will. Bye to DDoS attacks.

And at the end, a plaintive "We're Hiring..." Like everyone else here. Man I need some good Web ops people. I have two open spots. We're hiring too!!!

Question: You do lots of mini-code pushes (20/day). How the heck do you manage that and keep the site up? He says - culture is the biggest thing. They have devs that think like ops and don't do retarded things. They're ganglia addicted and they're the ones hitting the big red buttons. Then less important are some technical parts, like a one button deploy and verbose logging of changes.

He uses more dirty words than I do. Boss.

Artur Bergman of Wikia speaks again, on Squid vs Varnish.

PHP is a pig and wikitext is hard to parse, so they need caching. A hit is 8 ms and a miss is 200 ms, and they have a 75% hit rate. You have to get the cache hits up, by making more cachable. Ooo, they're playing around with ESI he says!

They decided to force caching for anonymous users. They've only gone up to 30 seconds, but no complaints. They ignore if-modified-since and purge. Be careful about vary-accept on encoding because there's an annoying browser bug with misplaced commas.

Mediawiki lives and dies by squid and puts cache control in the code, which is bad because developers are stupid.

Squid - the slide actually says "Me hates it" and "Still a piece of shit." Awesome.

Varnish - he loves varnish. He nearly cried when he read the source code (C). But it's a little unstable. He got it up to 65k hps (squid doing 2800).

Varnish has some "novel" techniques. Its control language is VCL. (Side note, they monitor with lvs). This gets compiled down to C at runtime. So you can put assembly in if you want. Lawdy. It segfaults from time to time under load and they're helping fix it. In a month or two he'll have it crackin'!

And the last one - Puppet, by Luke Kanies, Puppet developer.

Automation tools are old and bad, and especially because they're SSH based. (Agree!) And also because there's not many people who cross the chasm between sysadmin and developer. They decided they had to solve the problem and create something a billion times better than anything (where anything is cfengine). Either you can manage many machines with little effort, or you can't. You want to be able to. So this required abstraction. He's using the analogy of C scaring the bejeezus out of assembly programmers - a good analogy.

It's sad you have to do it, but he goes into why a more powerful tool should not scare people and put them all out of work. Developers seem to have gotten over this but not sysadmins. it's stupid especially because "we're understaffed" is the #1 thing I hear out of all of ours.

So they implemented Puppet with the metaphor of resources and resource providers, hiding all the file/command/UNIX admin stuff. (Well, kinda.) It's easily extensible.

The Web 2.0 crowd has made "microformats." Your infrastructure can use that idea too. Catch up with the times - if you're proud of doing something developers have been doing for 10 years (like moving to version control in Subversion) then you're behind. (Use git!) Anyway, you have to use polymorphism (overloading) to make a system like Puppet understand ssh on system 1 vs 2 vs 3.

Also, have one solution per problem. Not multiple. And most of the problems you face are NOT unique to you or your organization - so using a common tool like this can benefit from the network effect.

And the third big principle (were there only two before?) is completeness. Everything that matters in your config should be in the config. Not some minimal set. Relationships are important (dependencies). You can do things like have a service subscribe to a file and restart when it changes, for example.

Puppet is mainly used as a central config management tool. Each host gets a resource catalog. Machines get put in classes and they get lists of resources.

Puppet clients retrieve their resource catalog, determine order, check em, fix, and repeat every 30 minutes. "Like cfengine but sexier!" The completeness approach means clean management through the lifecycle - a freshly kickstarted box doesn't end up different. You just kickstart enough to run puppet and use it to do everything. So all boxes are kept 100% up to date without artifacts.

And it has reporting underway too! They're planning to charge for that to make some mooonay! Google, Stanford, Sony, Rackspace all use Puppet.

Why Puppet vs Capistrano? Cap is SSH in Ruby. Not something for yoru whole infrastructure.

Why Puppet vs cfengine? More open dev community and better.

What about Puppet slowness? It scales like HTTPS.

Puppet: Is XMLRPC but moving to REST. Uses certs and SSL, not keypairs. It's in Ruby. He's had to learn to be a developer in the process. It's also an API to the systems. It supports VMs well and can get into the guts of the VMs unlike pure VM provisioning tools. Buy me! it's open source but he sells support/trainin/addons. Discovery to come! There's nagios integration of some sort. Vertebra, like capistrano, is an ad hoc change tool - Puppet isn't (though you can use relsh for that).

That's the last session - wrapup later once I power up my laptop and get some booze in me!

First, it's the eagerly awaited Even Faster Web Sites. Steve Souders, previously Yahoo! performance guru and now Google performance guru, has another set of recommendations regarding Web performance. His previous book with its 14 rules and the Firebug plugin, YSlow, that supported it, are one of the things that really got us hooked deeply into the Web performance space.

First, he reviews why front end performance is so important. In the steady state, 80-90% of your average page's load time the user sees is time after the server has spit it out. "Network time." Optimizing your code speed is therefore a smaller area of improvement than optimizing the front end. And it can be improved, often in simple ways.

Man, there's a wide variance in how people's pages perform with a primed cache - from no benefit (most of the Alexa top 10) to incredible benefit (Google and MS live Search results pages). Anyway, Steve developed his original 14 best practices for optimizing front end performance, and then built YSlow to measure them.

And now, he's starting a second book! There's ten more best practices. The slides are on his blog if you want the full version.

• Split the initial payload
• Load scripts without blocking
• Don't scatter scripts
• Split dominant content domains
• Make static content cookie-free
• Reduce cookie weight
• Minify CSS
• Optimize images
• Use iframes sparingly
• To www or not to www

Neat tool he wrote - cuzillion, a page simulation tool.

"Splitting the payload" means only initially loading the JavaScript needed for page load, and lazy-loading everything else. On average, people only execute about 25% of their JS functions before the onLoad() fires. Tools to do this separation are needed (Microsoft's Doloto does but it's not public).

MSN.com figured out how to do parallel script downloads, a pretty simple but clever method using document.write, and showed it could be done. There's a mess of options - XHR eval, XHR injection, script in iframe, script DOM element, script defer, and document.write Script tag. Each approach has a bunch of gotchas. Some only work in IE, some require JS changes. Some trigger the various "busy" indicators (throbbers, statusbars) in the browsers. Then you need to figure out if you need guaranteed execution order or if you want "fastest" (execute as they get there). The "winner" is actually a decision tree, but boils down to Script DOM element or XHR injection. Kinda.

Inline scripts also have issues - they block rendering and downloads as well. There's some mitigations. Also, IE doesn't block on parallel CSS downloads unless they follow an inline script. So don't "scatter" scripts through your page, they screw things up.

So yay, front end performance... Me and my guys are starting to worry, however, about whether people are starting to ignore back end performance. Seems like a lot of these changes are getting pretty ticky and though our "best case" performance is mostly front end most of the performance *problems* are back end.

Having sweated my ass off for long enough, I displace to the larger ballroom for "Performance Plumbing" with Adam Bechtel from Yahoo!. He starts off talking about the natural progress of a growing site, from one server to a mess of them to two data centers to widely geo-distributed centers, where you start to run your own networks between them. And connect your network up with partners. Suddenly you have a global backbone. And you develop a bunch of routing policies.

You deploy "anycast" services. And scale them. This is dry. Very dry. How do you monitor your anycast service? Make caches. Proxies. Aside from humorous plumbing pictures I'm a little unclear what I'm supposed to be getting out of this talk. Luckily, it's over. Robert leans over to me and says, "I'm not sure what to say about that."

Back to the hot room for Stefan Stoyanov of Yahoo! about image size reduction. Seven tips. Again, can barely hear him (bad sound in this room + mumbling).

• Use png not gif.
• Run pngcrush.
• Remove JPEG metadata with jpgtran.
• Use palette not truecolor PNGs.
• There's an AlphaImageLoader CSS filter to help with the truecolor PNG alpha transparency problem in earlier IEs. Try not to use filters, use PNG8 or "the underscore hack _filter" (no, I don't know what this means either).
• Don't serve generated images as-is. Write generated images to disk and crush them. He compares some images form normal to crushed to PNG8, and I sure can't tell the difference, and you get like 50% savings at PNG8.
• Use CSS sprites and optimize them (all horizontal ideally) and close together. Stay within 256 colors and use PNG8.

Bonus:

• Don't serve images from your app server.

You should be able to save 10-30% on your image sizes. Man that went fast.

Next, shared dictionary compression over HTTP with Wei-Hsin Lee from Google. Called SDCH, or "Sandwich." Compression is nice, but if you could not retransmit identical pieces of data (like WAN accelerators do at the network level), you're cooking with gas. Good concept, I've always thought that ESI markup of pages was too manual and "my damn computer should figure that out by itself."

RFC 3229 for delta compression in HTTP - judged no good. Too many states to track and too limited scope (same URL only). Dame da!!!

So Google has introduced SDCH. It's an HTTP addition with a small set of states shared between client and server. It's domain scoped, like a cookie. It uses vcdiff (RFC 3284) plus enhancements, and it defines the server/client interaction wrt capabilities and dictionary.

Basically sdch is listed in the HTTP Accept-Encoding: variable. The server then sends a Get-Dictionary: response with the page, which the client lazy-loads. Then the client says what dictionaries it has (again, much like cookies) and the server responds with a sdch encoded response.

It complements gzip and deflate and generates a 40% improvement on google search results pages! But woirking on how to get it to the users. Spec and encoders and decoders are coming soon.

But it's unclear to me how this is better than ESI - he says that generating the dicts is site-specific, aka manual. I asked him during the question time but didn't get the question across to him I fear.

Building faster pages in Firefox and IE! Mike Connor (FF) and Lawrence/Stockwell (MS).

In Firefox 3 we used: dtrace (Sun), Shark (Apple), and talos (perf framework). Parallel conns up from 2 to 6 and stylesheet loading no longer blocks other downloads. (Seems like many of the Yahoo! High Performance Rules may be largely mitigated by browser fixes over time).

When working on perf - test. Test before and after. Measure continuously. Perf test after every checkin because cost to fix spikes later.

Easy steps: Use the Yahoo! guidelines (oo, he noted they're trying to make them not matter any more, I'm so smart), incremental layout, and avoid "quirks mode."

Hard steps: Use FF DOM Storage implementation. Optimize images with OptiPNG or pngcrush. Trim page markup site-wide.

Use tools. Firebug/YSlow, Jiffy. Shark on OSX. jQuery profiling plugin. [E: Oh good, that might help us with the jQuery-running-rampant I just discovered on our home page.]

Measure. Mozilla uses Talos to monitor perf against a static pageset. Can be set to use FF vs a live pageset. Regress!

In FF 3.1 looking to make JSON native and faster DOM implementation, and making Firebug rock more.

Would you like to know more?

On to IE. IE8: We're Fixing It! The first part was retread of the IE talk from yesterday. Some interesting things - XDomainRequest for cross-domain communication (really?) and adding a timeout to XMLHTTPRequest.

Now they're going to show off Performance Analyzer, which is built on top of ETW. Now they show us an internal plugin (DTW) that "wont' be released externally." Then why do I give a shit? I mean, come on.

They make a point that the various browser toolbars have ill effects. Add-ons of various sorts have the same problems. He has some tips for making well behaved add-ons, like use a separate thread.

Not a word of this had to do with building faster sites for IE. Boo. I was happy with the microsofties up till now, participation in the previous panels/sessions was good and on point, but this is a step back.

Finally a break. Afterwards, the glorious conclusion!

In the meantime, Peco had met our old friend Alistair Croll, once of Coradiant and now freelance, blogging on "Bitcurrent." Oh, and also at the vendor expo yesterday we saw something exciting - an open source offering from a company called ControlTier, which is a control and deployment app. We have one in house largely written by Peco called "Monolith" - more for control (self healing) and app deploys, which is why we don't use cfengine or puppet, which have very different use cases. His initial take is that ControlTier has all the features he's implemented and all the ones on his list to implement for Monolith, so we're very intrigued.

We kick off with a video of base jumpers, just to get the adrenaline going. Then, a "quirkily humorous" video about Faceball.

Steve and Jesse kick us off again today, and announce that the conference has more than 600 attendees, which is way above predictions! Sweet. And props to the program team, Artur Bergman (Wikia), Cal Henderson (Yahoo!), Jon Jenkins (Amazon), and Eric Shurman (Microsoft). Velocity 2009 is on! This makes us happy, we believe that this niche - web admin, web systems, web operations, whatever you call it - is getting quite large and needs/deserves some targeted attention.

First up is the open data talk from Ignite! last night; as the winner she is re-presenting. Go back and read my previous post for more, since I've heard it already.

EUCALYPTUS, by Rich Wolski, is an acronym for "Elastic Utility Computing Architecture for Linking Your Programs to Useful Systems." There's a bunch of "cloud" solutions in the commercial space, from Amazon to 3tera to mosso. But as this technology emerges, the question comes up of haw it can be adapted to other uses- like scientific/academic, data assimilation, gaming, mobile... Some things work great with a grid as currently conceived but some don't.

Note to NI: the data assimilation arena is something we should be more involved in. It's core to our business as being all about test and measurement. See the new issue of Wired for more.

So they looked at nimbus, an interface to globus, and old reliable grid thing. And Enomalism. But neither fit their need. So they made Eucalyptus.

• Strict Web services
• Linux based, and using an interface designed to be compatible with Amazon EC2
• Low impact on installations, one-button install using Rocks (since SAs rare in academia)

Oh, I get it now. It's an open source tool to encourage research in elastic/cloud/utility computing. So essentially a cloud "development environment." Experiment with it, test for free, then move to a cloud.

They also have requirements the current clouds don't serve well - like accountability of userids. And more. So I'm as "excited" about cloud computing as the next guy but I (and the people around me, it seems) have heard about enough of it for the week. He goes on at some length about the spiffy things Eucalyptus does that helps out people in real environments (IP scarcity, EC2 compatibility, admin roles, etc.) And they created "Rocks," a cloud friendly one button install. Not sure what features it has that aren't supported by puppet or whatnot - VM support? They use Mule, the open source ESB.

Lessons learned: A fully open source cloud was harder than it should be. And admins still seem to do lots of things by hands than using Rocks, let alone a cloudwide provisioning tool. They are planning to integrate with RightScale, and add VMWare supprot (just Xen now). So yay to your NSF dollars at work!

Next we've got Harold Prokop from Akamai. We're interested because we're doing a CDN evaluation at NI this year. "We didn't really think of Akamai as a cloud, but since cloud is the new hot term, it's a cloud!" That's paraphrased from his talk, but not much.

The problem with traditional CDNs is that as more content is dynamic, less of it is cacheable. (Well, kinda.) So they talk about their new approach of "better routing" - since they have POPs everywhere, they can essentially do better-than-BGP routing and use WAN acceleration techniques (like not using the fat old bitch that is TCP) to speed up traffic across the Net. It's called "Akamai SureRoute." Seems like a good idea really. And intelligent prefetching. He also talks about ESIs (Edge Side Includes), a good idea- we've looked at them, as Oracle WebCache supports it too - to mark up parts of a page with differential cacheability rules. The thing that scares me about ESI is that no one else has ever picked up on this open standard; it's been Oracle + Akamai for like 10 years, though it is a W3C standard.

How about moving business logic to the edge? Hmmm, sounds good but I wonder how realistic that is - you really have to factor your code carefully for that. But if you're really into assembling from mini-components/services it could work.

Question: But we're using Ajax and JavaScript - why do we need this? His answer is more diplomatic than mine, which is "you can't implement much worthwhile in JavaScript." For anyone who's not a pure Web 2.0 play, that is. And Ajax is about communicating back with the server so then all this applies anyway, which means the question was more just keyword gibberish than well thought through.  I mentally try to set the questioner ablaze but it doesn't work.

Then there was a large break where we were supposed to go to the vendor floor, but instead we talked at length with Akamai and Coradiant. We're looking into CDNs for ni.com and hence Akamai, and we're Coradiant customers. We've had a kinda long and trial-filled path in getting our Coradiant device working; part of the problems were ours (hint: use taps not span ports) and part the product's, but the bottom line is it's not giving us good data yet and that makes the guy signing the checks grumpy.  But they seem committed to getting us more engaged with them, getting on the advisory board, etc. That'll be good - I think the RUM space and especially the network-sniff sort to be a very powerful tool in the APM suite and we're making a strategic shift to it from synthetic for more granular managing of app performance with our developer teams.

The next segment is a whole set of demos of various nifty tools.

HttpWatch is demoed by Simon Perkins. It's a HTTP sniffer and viewer. It shows you a waterfall for a request/response like YSlow or Page Detailer, but it's pretty. The free version I just downloaded only shows the waterfall but you can't drill down into a component - looks like the full version is $300. I wonder if it's $300 better than the others though. And it's hard to tell if the free downloadable is feature-crippled; I consider that bad form and prefer time-crippled. Now that there's like 20 of these tools it's time for a solid bakeoff.

Eric Lawrence demoes Fiddler. He works on IE and developed TamperIE, Fiddler, etc. Fiddler's more of a "platform for debugging" than a tool like HttpWatch, he says. It runs as a local proxy. They developed it to fix Microsoft's Web site. And it simulates lower bandwidth connections! And you can modify traffic. So it's a very cool toolkit for more generalized work!

Eric Goldsmith of AOL shows AOL PageTest. It's an open source IE plugin - or even you can use it online at www.webpagetest.org. Again, a waterfall. It also has optimization tips built in. One note - the browser version doesn't have a legend in it anywhere for the colors, which is obnoxious (the web page version has a legend at the top).

Firebug in the hizzouse! John J. Barton shows it off. For the two of you who haven't used it, it's a Firefox plugin that lets you inspect/debug all kinds of stuff, especially JavaScript, CSS, and the DOM. Then you can profile the page. This'll show you every JS call and how much time it took. Running this on the ni.com home page shows an unhealthy amount of junk from jquery, I note. But it all got over my head quick. I'm no JavaScript programmer.

Last for the morning, Sean Quinlan from Google talks about Storage at Scale. Their philosophy is very low end PC based - single machine performance isn't interesting because partitioning problems is easy and even expensive hardware fails, so they build reliability into software not hardware. They've built GFS, a non-POSIX clustered file system, and Bigtable, a distributed non-SQL database. They need ginormous scaling of course.

Ernest thinks, "I've never understood why our server teams at NI insist on more expensive hardware - even after moving from Sun to Dell they insist on everything being dual power supply, RAID 5, etc. That's why we have clusters. I want cheaper hardware!"

GFS spreads chunks of files out and duplicates chunks across servers. Fault tolerance is their largest concern - at scale commodity hardware is failing a lot. Checksumming and replication work well against this. He gets into details way too complex to reproduce here, if you care go look up GFS, there's a paper on it.

Bigtable was "database-like" and built on top of GFS. More of a massive hash really, key/value pairs. They needed a way to keep semi-structured data at scale.

Lunch beckons and my butt is numb.