<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Web Admin Blog &#187; Monitoring</title>
	<atom:link href="http://www.webadminblog.com/index.php/category/monitoring/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.webadminblog.com</link>
	<description>Real Web Admins.  Real World Experience.</description>
	<lastBuildDate>Thu, 22 Jul 2010 16:18:30 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.0.1</generator>
		<item>
		<title>Enterprise Systems vs. Agility</title>
		<link>http://www.webadminblog.com/index.php/2010/02/09/enterprise-systems-vs-agility/</link>
		<comments>http://www.webadminblog.com/index.php/2010/02/09/enterprise-systems-vs-agility/#comments</comments>
		<pubDate>Tue, 09 Feb 2010 22:06:18 +0000</pubDate>
		<dc:creator>Ernest</dc:creator>
				<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[agility]]></category>
		<category><![CDATA[architecture]]></category>
		<category><![CDATA[systems]]></category>

		<guid isPermaLink="false">http://www.webadminblog.com/?p=396</guid>
		<description><![CDATA[I was recently reading a good Cameron Purdy post where he talks about his eight theses regarding why startups or students can pull stuff off that large enterprise IT shops can't. My summary/trenchant restatement of his points: Changing existing systems is harder than making a custom-built new one (version 2 is harder) IT veterans overcomplicate [...]]]></description>
			<content:encoded><![CDATA[<p>I was recently reading a <a href="http://www.jroller.com/cpurdy/entry/eight_theses">good Cameron Purdy post</a> where he talks about his eight theses regarding why startups or students can pull stuff off that large enterprise IT shops can't.</p>
<p>My summary/trenchant restatement of his points:</p>
<ol>
<li>Changing existing systems is harder than making a custom-built new one (version 2 is harder)</li>
<li>IT veterans overcomplicate new systems</li>
<li>The complexity of a system increases exponentially the work needed to change it (versions 3 and 4 are way way harder)</li>
<li>Students/startups do fail a lot, you just don't see those</li>
<li>Risk management steps add friction</li>
<li>Organizational overhead (paperwork/meetings) adds friction</li>
<li>Only overconservative goons work in enterprise IT anyway</li>
<li>The larger the org, the more conflict</li>
</ol>
<p>Though I suspect #1 and #3 are the same, #2 and #5 are the same, and #6 and #8 are the same, really.</p>
<p>I've been thinking about this lately with my change from our enterprise IT Web site to a new greenfield cloud-hosted SaaS product in our R&amp;D organization.  It's definitely a huge breath of fresh air to be able to move fast.  My observations:</p>
<h2>Complexity</h2>
<p>The problem of systems complexity (theses #1 and #3) is a very real one.  I used to describe our Web site as having reached "system gridlock."  There were hundreds of apps running dozens to a server with poorly documented dependencies on all kinds of stuff.  You would go in and find something that looked "wrong" - an Apache config, script, load balancer rule, whatever - but if you touched it some house of cards somewhere would come tumbling down.  Since every app developer was allowed to design their own app in its own tightly coupled way, we had to implement draconian change control and release processes in an attempt to stem the tide of people lining up to crash the Web site.</p>
<p>We have a new system design philosophy for our new gig which I refer to as "sharing is the devil."  All components are separated and loosely coupled.  Using cloud computing for hardware and open source for software makes it easy and affordable to have a box that does "only one thing."  In traditional compute environments there's pressure to "use up all that CPU before you add more", which results in a penny wise, pound foolish strategy of consolidation.  More and more apps and functions get crunched closer together and when you go back to pull them out you discover that all kinds of new connections and dependencies have formed unbidden.</p>
<h2>Complication</h2>
<p>Overcomplicating systems (#2 and #5) can be somewhat overcome by using agile principles.  We've been delving heavily into doing not just our apps but also our infrastructure according to an agile methodology.  It surfaces your requirements - frankly, systems people often get away with implementing whatever they want, without having a spec let alone one open to review.  Also, it makes you prioritize.  "Whatever you can get done in this two week iteration, that's what you'll have done, and it should be working."  It forces focus on what is required to get things to work and delays more complex niceties till later as there's time.</p>
<h2>Conservatism</h2>
<p>Both small and large organizations can suffer from #6 and #8.  That's mostly a mindset issue.  I like to tell the story about how we were working on a high level joint IT/business vision for our Web site.  We identified a number of "pillars" of the strategy we were developing - performance, availability, TCO, etc.  I had identified agility as one, but one of the application directors just wasn't buying into it.  "Agility, that's weird, how do we measure that, we should just forget about it."  I finally had to take all the things we had to the business head of the Web and say "of these, which would you say is the single most important one?"  "Agility, of course," he said, as I knew he would.  I made it a point to train my staff that "getting it done" was the most important thing, more important than risk mitigation or crossing all the t's and dotting all the i's.  That can be difficult if the larger organization doesn't reward risk and achievement over conservatism, but you can work on it.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.webadminblog.com/index.php/2010/02/09/enterprise-systems-vs-agility/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>A DoS We Can Believe In</title>
		<link>http://www.webadminblog.com/index.php/2009/01/21/a-dos-we-can-believe-in/</link>
		<comments>http://www.webadminblog.com/index.php/2009/01/21/a-dos-we-can-believe-in/#comments</comments>
		<pubDate>Wed, 21 Jan 2009 20:58:46 +0000</pubDate>
		<dc:creator>Ernest</dc:creator>
				<category><![CDATA[Application Performance Management]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[internet]]></category>
		<category><![CDATA[network]]></category>
		<category><![CDATA[obama]]></category>

		<guid isPermaLink="false">http://www.webadminblog.com/?p=171</guid>
		<description><![CDATA[We knew that the historic inauguration of Barack Obama would be generating a lot more Internet traffic than usual, both in general and specifically here at NI.  Being prudent Web Admin types, we checked around to make sure we thought that there wouldn't be any untoward effects on our Web site.  Like many corporate sites, [...]]]></description>
			<content:encoded><![CDATA[<p>We knew that the historic inauguration of Barack Obama would be generating a lot more Internet traffic than usual, both in general and specifically here at NI.  Being prudent Web Admin types, we checked around to make sure we thought that there wouldn't be any untoward effects on our Web site.  Like many corporate sites, we use the same pipe for inbound Internet client usage and outbound Web traffic, so employees streaming video to watch the event could pose a problem.  We got all thumbs up after consulting with our networking team, and decided to not even send any messaging asking people to avoid streaming.  But, we monitored the situation carefully as the day unwound.  Here's what we saw, just for your edification!</p>
<p>Our max inbound Internet throughput was 285 Mbps, about double our usual peak.  We saw a ni.com Web site performance degradation of about 25% for less than two hours according to our Keynote stats.  ni.com ASPs were affected proportionately which indicates the slowdown was Internet-wide and not unique to our specific Internet connection here in Austin.  The slowdown was less pronounced internationally, but still visible.  So in summary - not a global holocaust, but a noticeable bump.</p>
<p>Cacti graphs showing our Internet connection traffic:</p>
<p><img class="aligncenter size-full wp-image-172" title="obamabumpcactihrly" src="http://www.webadminblog.com/wp-content/uploads/2009/01/obamabumpcactihrly.png" alt="obamabumpcactihrly" width="591" height="257" /><img class="aligncenter size-full wp-image-173" title="obamabumpcactidaily" src="http://www.webadminblog.com/wp-content/uploads/2009/01/obamabumpcactidaily.png" alt="obamabumpcactidaily" width="591" height="257" /></p>
<p>Keynote graph of several of our Web assets, showing global response time in seconds:<img class="aligncenter size-full wp-image-174" title="obamabumpkeynote" src="http://www.webadminblog.com/wp-content/uploads/2009/01/obamabumpkeynote.png" alt="obamabumpkeynote" width="800" height="500" />Looking at the traffic specifically, there were two main standouts.  We had TCP 1935, which is Flash RTMP, peaking around 85 Mbps, and UDP 8247, which is a special CNN port (they use a plugin called "Octoshape" with their Flash streaming), peaking at 50 Mbps.   We have an overall presence of about 2500 people here at our Austin HQ on an average day, but we can't tell exactly how many were streaming.  (Our NetQoS setup shows us there were 13,600 'flows,' but every time a stream stops and starts that creates a new one - and the streams were hiccupping like crazy.  We'd have to do a bunch of Excel work to figure out max concurrent, and have better things to do.)</p>
<p>In terms of the streaming provider breakdown - since everyone uses Akamai now, the vast majority showed as "Akamai".  We could probably dig more to find out, but we don't really care all that much.  And, <a href="http://www.techcrunch.com/2009/01/21/the-day-live-web-video-streaming-failed-us/" target="_blank">many of the sources were overwhelmed</a>, which helped some.</p>
<p>We just wanted to share the data, in case anyone finds it helpful or interesting.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.webadminblog.com/index.php/2009/01/21/a-dos-we-can-believe-in/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>
