Web Admin Blog Real Web Admins. Real World Experience.

8May/140

Analyzing NetFlow for Data Loss Detection

The 2014 Verizon Data Breach Investigation Report (DBIR) is out and it paints quite the gloomy picture of the world we live in today where cyber security is concerned.  With over 63,000 security incidents and 1,367 confirmed data breaches, the question is no longer if you get popped, but rather, when.  According to the report, data export is second only to credit card theft on the list of threat actions as a result of a breach.  And with the time to compromise typically measured in days and time to discovery measured in weeks or months, Houston, we have a problem.

I've written in the past about all of the cool tricks we've been doing to find malware and other security issues by performing NetFlow analysis using the 21CT LYNXeon tool and this time I've found another trick around data loss detection that I thought was worth writing about.  Before I get into the trick, let's quickly recap NetFlow for those who aren't familiar with it.

Think of NetFlow as the cliff notes of all of the network traffic that your systems handle on a daily basis.  Instead of seeing WHAT data was transmitted (a task for deep packet inspection/DPI), we see the summary of HOW the data was transmitted.  Things like source and destination IP, source and destination port, protocol, and bytes sent and received.  Because many network devices are capable of giving you this information for free, it only makes sense to capture it and start using it for security analytics.

So, now we have our NetFlow and we know that we're going to be breached eventually, the real question becomes how to detect it quickly and remediate before a significant data loss occurs.  Our LYNXeon tool allows us to create patterns of what to look for within NetFlow and other data sources.  So, to help detect for data loss, I've designed the following analytic:

LYNXeon Analytics for Data Loss

What this analytic does is it searches our NetFlow for any time an internal IP address is talking to an external IP address.  Then, it adds up the bytes sent for each of these unique sets of connections (same source, destination, and port) and presents me with a top 25 list.  Something like this:

Top 25 List

So, now we have a list of the top 25 source and destination pairs that are sending data outside of our organization.  There are also some interesting ports in this list like 12547, 22 (SSH), 443 (HTTPS), and 29234.  A system with 38.48 GB worth of data sent to a remote server seems like a bad sign and something that should be investigated.  You get the idea.  It's just a matter of analyzing the data and separating out what is typical vs what isn't and then digging deeper into those.

My advice is to run this report on an automated schedule at least daily so that you can quickly detect when data loss has begun in order to squash it at the source.  You could probably argue that an attacker might take a low and slow approach to remain undetected by my report, and you'd probably be right, but I'd also argue that if this were the case, then I've hopefully slowed them enough to catch them another way within a reasonable timespan.  Remember, security is all about defense in depth and with the many significant issues that are highlighted by the Verizon DBIR, we could use all of the defense we can muster.

3Sep/130

Six Reasons Why Your Company Needs a Chief Information Security Officer (CISO)

I am going to start out here by saying that I do not now, nor have I ever, held the title of Chief Information Security Officer (CISO).  That having been said, I do effectively fill this role as the Information Security Program Owner for a large, $1B+ per year, public company.  Some of what follows will be diatribe on my current role and what I would change if given the opportunity.  Some of it will be based on general observations of how I've seen other companies handle internal security.  What follows are six reasons why your company needs a Chief Information Security Officer (CISO).

Let's start out with how I got my current title.  Early in my career I was a *nix Administrator working for a number of different companies.  I did everything from working as support at a website hosting company to building systems as a military contractor.  Even though my official title never had anything to do with security, I have always had a passion for it, so I always found a way to make it part of my job.  Fast forward to about seven years ago where I got a job as a Web Systems Engineer at my current employer.  I quickly realized that there was nobody handling security for our systems so I decided to shoulder that responsibility.  I began by running Qualys scans, analyzing the results, and fixing the vulnerabilities.  Since this was not my primary role, it all of this work was performed in about 5% of my overall time spent, but I was able to keep good metrics and show fantastic results over time.  After several years of working like this, I finally made the decision to dedicate myself to security full-time.  I got a job offer from another company to be a Security Engineer, but decided to see if my current company was interested in allowing me a similar move.  After some discussions and a few presentations on what the job would entail, I was officially allowed to spend 100% of my time on security.  The only catch was that I was now responsible for our IT SOX testing as well.  Now came the question of what to call my new role.  The title "C" anything is reserved for our executives so that took CISO off the table immediately.  Sad.  I was also not allowed to use the term "Manager" in my title since that indicated having people underneath me.  Even sadder.  I finally settled on "Information Security Program Owner" as it indicated an ownership role in security (as close to Manager as I could get) while staying away from those other non-sanctioned titles.

Alright, so what does any of that have to do with needing a CISO you ask?  To start with, I was the only security professional in the entire company of roughly 5,000 employees at the time.  While officially my purview was in the area of IT security for the enterprise, the lack of any other experts quickly made me a hot commodity.  I was asked to participate on various architecture teams, several teams having to do with regulatory compliance, and even to consult with our R&D teams on product security from time to time.  I'd like to believe that it was because I am so awesome that people couldn't get enough of me, but the God's honest truth is that the entire company had a need and desire for security and there wasn't anyone else to assist.  Which leads me to the first reason why your company needs a Chief Information Security Officer:

Reason #1: By definition, the CISO is where the buck stops as far as security is concerned for your organization.  It is the CISO's job to make sure that security is a concerted effort and that your efforts are not inefficiently duplicated in multiple business units.  Without a CISO, you may may have operational security, but you likely lack direction or a long-term plan for an actual security program.

Now, while my title says "he owns the security program", the fact is that I am not officially a manager or executive.  Thus, on an official level, I pull about as much weight as any other individual contributor in the organization.  It's a precarious position to be in.  On one hand I'm charged with ensuring the security of everyone and everything in the company.  Sometimes this can require being the bad guy and telling people their stuff is broken.  On the other hand, I don't hold enough power to actually force any action that others don't actually want to take.  Perhaps I'll write a future post about how I've managed to still get things done despite this dilemma, but for now this leads me to the second reason why your company needs a Chief Information Security Officer:

Reason #2: Designating one of your senior security resources as the CISO is a form of empowerment.  You are making a statement that they are the person that you trust to make informed security decisions for the organization.  It helps if you can have them report to another C-level executive, like the CFO, but the most important thing here is the title as Chief Information Security Officer says that they are in charge of everything security (everything Information Security if you want to get technical) for your organization.  This helps tremendously in ensuring that security is still a priority when business turns political.

When you hear the title Chief Information Security Officer, what do you think of?  Maybe the IT guy who handle the IPS system?  The guy who goes running around when a system is infected with malware?  Maybe even the guy who wrote the Information Security Policy if you're lucky?  Your CISO should be all these things and so much more.  This leads me to the third reason why your company needs a Chief Information Security Officer:

Reason #3: Your CISO is all things security.  Wikipedia does a great job listing some of the many roles of the Chief Information Security Officer so I'm just going to steal them and list them here:

  • Information Security and Information Assurance
  • Information Regulatory Compliance (PCI, SOX, HIPAA, etc)
  • Information Risk Management
  • Supply Chain Risk Management
  • Cybersecurity
  • Information Technology Controls
  • Information Privacy
  • Computer Emergency Response Team
  • Identity and Access Management
  • Security Architecture
  • IT Investigations, Digital Forensics, and eDiscovery
  • Disaster Recovery and Business Continuity Management
  • Information Security Operations Center
  • PR

Obviously one person cannot handle all of these things which is why most companies have a team of security professionals (ie. Information Security Officers) who report up to the CISO, but this should give you an idea as to the wide scope of what the CISO is responsible for.  Chances are that if you don't have a CISO, then many of these activities aren't happening.  Even worse, the ones that are happening likely aren't aligned with your business objectives.  It's tough to justify spending any money on a program when it performs activities ad-hoc and completely separate from your business.  Which leads me to the fourth reason why your company needs a Chief Information Security Officer:

Reason #4: Your CISO is a business executive that spans into the technical world of security as well.  They should be involved in the business decisions of the company so that they can ensure that the company's security activities are well-aligned with the projects that the business is undertaking.

Hopefully, your senior security professional is an extremely valued member of your team.  If you are holding off on giving them an official Chief Information Security Officer title, then you are doing both them and your company a disservice.  Security companies are organizing events all the time that are targeted at these executives who control the security purse strings.  Sometimes they call them CISO Roundtables, Summits, or otherwise, but the gist of it is that they are a form of education for the CISO and provides them with the opportunity to network with other security professionals in the area, all on somebody else's dime.  The catch is that you're only invited if you're a CISO.  This leads me to the fifth reason why your company needs a Chief Information Security Officer:

Reason #5: The title of CISO is synonymous with "the person in charge of security" for your company and worlds of opportunity open up for them when you bestow upon them that title.  It means free lunches, free trainings, and a host of other perks that unfortunately aren't available with a title like "Security Manager" or "Senior Security Engineer".  Think of it as a job perk that doesn't cost your company a thing.

Before I wrap this up, I have one final reason why your company needs a Chief Information Security Officer, but it's certainly not for everyone.  Occasionally, you'll find a person both technically talented as well as someone who has an affinity and desire to do public speaking.  If this is your senior security person, then it's time to lock them down as they have the ability to do more positive marketing for your company than your entire marketing department.  This leads me to the sixth and final reason why your company needs a Chief Information Security Officer:

Reason #6: If your CISO is willing and able to give engaging talks about security-related topics, then that person, with that title, can make a world of difference for your organization from a marketing perspective.  Conferences are always looking for new and interesting talks and attendees often consider the speakers as industry luminaries.  No marketing whitepaper will ever come close to the exposure potential of having your own industry expert, presenting on a fantastic topic, using a company branded slide deck, in front of hundreds of security professionals.

There you have my six reasons why your company needs a Chief Information Security Officer.  I hope that this was helpful in your search for becoming or designating your company's ultimate CISO.  Feel free to add your own thoughts in the comments below.

23Jun/100

Looking for DevOps Stuff?

If you heard about us at Velocity 2010 and are coming here for sweet sweet DevOps and agile info,  we've moved to a different blog - come see us at the agile admin!

29Apr/100

Our First DevOps Implementation

Although we're currently engaged in a more radical agile infrastructure implementation, I thought I'd share our previous evolutionary DevOps implementation here (way before the term was coined, but in retrospect I think it hits a lot of the same notes) and what we learned along the way.

Here at NI we did what I'll call a larval DevOps implementation starting about seven years ago when I came and took over our Web Systems team, essentially an applications administration/operations team for our Web site and other Web-related technologies.  There was zero automation and the model was very much "some developers show up with code and we had to put it in production and somehow deal with the many crashes per day resulting from it."  We would get 100-200 on-call pages a week from things going wrong in production.  We had especially entertaining weeks where Belgian hackers would replace pages on our site with French translations of the Hacker's Manifesto.  You know, standard Wild West stuff.  You've been there.

Step One: Partner With The Business

First thing I did (remember this is 2002), I partnered with the business leaders to get a "seat at the table" along with the development managers.  It turned out that our director of Web marketing was very open to the message of performance, availability, and security and gave us a lot of support.

This is an area where I think we're still ahead of even a lot of the DevOps message.  Agile development carries a huge tenet about developers partnering side-by-side with "the business" (end users, domain experts, and whatnot).  DevOps is now talking about Ops partnering with developers, but in reality that's a stab at the overall more successful model of "biz, dev, and ops all working together at once."

28Apr/100

dev2ops Interview

Want to hear me spout off more about DevOps?  Well, here's your chance; I did an interview with Damon Edwards of DTO and they've posted it on the dev2ops blog!

Killer quote:

"I say this as somebody who about 15 years ago chose system administration over development.  But system administration and system administrators have allowed themselves to lag in maturity behind what the state of the art is. These new technologies are finally causing us to be held to account to modernize the way we do things.  And I think that’s a welcome and healthy challenge."

Tagged as: , No Comments
15Mar/100

P.S. We’re Hiring

Web Operations Engineer

If you're from the Austin area you probably know National Instruments - our campus is up here at Mopac and Braker; we've been named one of Fortune Magazine's 100 Best Companies To Work For eleven years running.

We've got an opening on a new team building cloud-based Web systems for new SaaS products and Web integration features of our existing hardware and software products.

We need someone to form the core of a new international Web Operations team to provide 24x7 support of these products. You'll work interactively with our R&D engineers, Web programmers, systems engineers, product support organization, and a host of other groups to this end.  You'll also help build up the operational environment - monitoring, provisioning, release process, high availability, reporting, performance assurance, security auditing, that kind of thing.  You’ll need to be able to document well and train other operations staff.

You can read the "straight" job posting and apply at ni.com/jobs under R&D as "Web Systems Engineer."

Benefits:

  • Work with smart people
  • Fun, fast-paced environment
  • We'll pay you in real money, not WoW gold
  • We're growing and profitable ::cough no layoffs cough::
  • Room to innovate

You should be skilled in administration of Linux and Windows systems (we run about 50/50), Java and .NET app servers, Web services, search, security, performance, high availability, infrastructure automation, and all the other fun things that Web systems people do.

This is a Web systems administration position with responsibility for realtime operations - it's not Web design, Web applications programming, help desk/end user support, or "back room" systems admin. If you don't want to spend your day configuring load balancing software, figuring out why someone's app is crashing the app server, finding security holes in a Web application, debugging why a new Web page is really slow, getting pages from server monitors, and coding up a great new tool to automate your team's work, this isn't the position for you.  There is oncall “pager duty” and some nights/weekends required.

Sorry, no recruiters or H1-B candidates.

12Mar/104

Before DevOps, Don’t You Need OpsOps?

From the "sad but true" files comes an extremely insightful point apparently discussed over beer by the UK devops crew recently - that we are talking about dev and ops collaboration but the current state of collaboration among ops teams is pretty crappy.

This resonates deeply with me.  I've seen that problem in spades.  I think in general that a lot of the discussion about the agile ops space is too simplistic in that it seems tuned to organizations of "five guys, three of whom are coders and two of whom are operations" and there's no differentiation.  In real life, there's often larger orgs and a lot of differentiation that causes various collaboration challenges.  Some people refer to this as Web vs Enterprise, but I don't think that's strictly true; once your Web shop grows from 5 guys to 200 it runs afoul of this too - it's a simple scalability and organizational engineering problem.

As an aside, I don't even like the "Ops" term - a sysadmin team can split into subgroups that do systems engineering, release management, and operational support...  Just saying "Ops" seems to me to create implications of not being a partner in the initial design and development of the overall system/app/service/site/whatever you want to call it.

Ops Verticals

Here, we have a large Infrastructure department.  Originally, it was completely siloed by technology verticals, and there's a lot of subgroups.  Network, UNIX, Windows, DBA, Lotus Notes, Telecom, Storage, Data Center...  Some ten plus years ago when the company launched their Web site in earnest, they quickly realized that wasn't going to work out.  You had the buck-passing behavior described in the blog posts above that made issues impossible to solve in a timely fashion, plus it made collaboration with devs/business nearly impossible.  Not only did you need like 8 admins to come involve themselves in your project, but they did not speak similar enough languages - you'd have some crusty UNIX admin yelling "WHAT ABOUT THE INODES" until the business analyst started to cry.

Dev Silos

But are our developers here better off?  They are siloed by business unit.  Just among the Web developers there's the eCommerce developers, eCRM, Product Advisors, Community, Support, Content Management...  On the one hand, they are able to be very agile in creating solutions inside their specific niche.  On the other hand, they are all working within the same system environment, and they don't always stay on the same page in terms of what technologies they are using. "Well, I'm sure THAT team bought a lovely million dollar CMS, but we're going to buy our own different million dollar CMS.   No, you don't get more admin resource."  Over time, they tried to produce architecture groups and other cross-team initiatives to try to rein in the craziness, with mixed but overall positive results.

Plugging the Dike

What we did was create a Web Administration group (Web Ops, whatever you want to call it) that was holistically responsible for Web site uptime, performance, and security.  Running that team was my previous gig, did it for five years.  That group was more horizontally focused and would serve as an interface to the various technology verticals; it worked closely with developers in system design during development, coordinated the release process, and involved devs in troubleshooting during the production phase.

BizOps?

In fact, we didn't just partner with the developers - we partnered with the business owners of our Web site too, instead of tolerating the old model of "Business collaborates with the developers, who then come and tell ops what to do."  This was a remarkably easy sell really.  The company lost money every minute the Web site was down, and it was clear that the dev silos weren't going to be able to fix that any more than the ops silos were.  So we quickly got a seat at the same table.

Results

This was a huge success.  To this day, our director of Web Marketing is one of the biggest advocates of the Web operations team.  Since then, other application administration (our word for this cross-disciplinary ops) teams have formed along the same model.  The DevOps collaboration has been good overall - with certain stresses coming from the Web Ops team's role as gatekeeper and process enforcement.  Ironically, the biggest issues and worst relationships were within Infrastructure between the ops teams!

OpsOps - The Fly In The Ointment

The ops team silos haven't gone down quietly.  To this day the head DBA still says "I don't see a good reason for you guys [WebOps] to exist."  I think there's a common "a thing is just the sum of its parts" mindset among admins for whatever reason.  There are also turf wars arising from the technology silo division and the blurring of technology lines by modern tech.  I tried again and again to pitch "collaborative system administration."  But the default sysadmin behavior is to say "these systems are mine and I have root on them.  Those are your systems and you have root on them.  Stay on your side of the line and I'll stay on mine."

Fun specific Catch-22 situations we found ourselves in:

  • Buying a monitoring tool that correlates events across all the different tiers to help root-cause production problems - but the DBAs refusing to allow it on "their" databases.
  • Buying a hardware load balancer - we were going to manage it, not the network team, and it wasn't a UNIX or Windows server, so we couldn't get anyone to rack and jack it (and of course we weren't allowed to because "Why would a webops person need server room access, that's what the other teams are for").

Some of the problem is just attitude, pure and simple.  We had problems even with collaboration inside the various ops teams!  We'd work with one DBA to design a system and then later need to get support from another DBA, who would gripe that "no one told/consulted them!"  Part of the value of the agile principles that "DevOps" tries to distill is just a generic "get it into your damn head you need to be communicating and working together and that needs to be your default mode of operation." I think it's great to harp on that message because it's little understood among ops.  For every dev group that deliberately ostracizes their ops team, there's two ops teams who don't think they need to talk to the devs - in the end, it's mostly our fault.

Part of the problem is organizational.  I also believe (and ITIL, I think, agrees with me) that the technology-silo model has outlived its usefulness.  I'd like to see admin teams organized by service area with integral DBAs, OS admins, etc.  But people are scared of this for a couple reasons.  One is that those admins might do things differently from area to area (the same problem we have with our devs) - this could be mitigated by "same tech" cross-org standards/discussions.  The other is that this model is not the cheapest.  You can squeeze every last penny out if you only have 4 Windows admins and they're shared by 8 functional areas.  Of course, you are cutting off your nose to spite your face because you lose lots more in abandoned agility, but frankly corporate finance rules (minimize G&A spending) are a powerful driver here.

If nothing else, there's not "one right organization" - I'd be tempted to reorg everyone from verticals into horizontals, let that run for 5 years, and then reorg back the other way, just to keep the stratification from setting in.

Specialist vs Generalist

One other issue.  The Web Ops team we created required us to hire generalists - but generalists that knew their stuff in a lot of different areas.  It became very hard to hire for that position and training took months before someone was at all effective.  Being a generalist doesn't scale well.  Specialization is inevitable and, indeed, desirable (as I think pretty much anything in the history of anything demonstrates).  You can mitigate that with some cross-training and having people be generalists in some areas, but in the end, once you get past that "three devs, two ops, that's the company" model, specialization is needed.

That's why I think one of the common definitions of DevOps - all ops folks learning to be developers or vice versa - is fundamentally flawed.  It's not sustainable.  You either need to hire all expensive superstars that can be good at both, or you hire people that suck at both.

What you do is have people with varying mixes.  In my current team we have a continuum of pure ops people, ops folks doing light dev, devs doing light ops, and pure devs.  It's good to have some folks who are generalizing and some who are specializing.  It's not specializing that is bad, it's specialists who don't collaborate that are bad.

Conclusion

So I've shared a lot of experiences and opinions above but I'm not sure I have a brilliant solution to the problem.  I do think we need to recognize that Ops/Ops collaboration is an issue that arises with scale and one potentially even harder to overcome than Dev/Ops collaboration.  I do think stressing collaboration as a value and trying to break down organizational silos may help.  I'd be happy to hear other folks' experiences and thoughts!

11Mar/101

Defining Agile Operations and DevOps

I recently read a great blog post by Scott Wilson that was talking about the definitions of Agile Operations, DevOps, and related terms.  (Read the comments too, there's some good discussion.)  From what I've heard so far, there are a bunch of semi-related terms people are using around this whole "new thing of ours."

The first is DevOps, which has two totally different frequently used definitions.

1.  Developers and Ops working closely together - the "hugs and collaboration" definition

2.  Operations folks uptaking development best practices and writing code for system automation

The second is Agile Operations, which also has different meanings.

1.  Same as DevOps, whichever definition of that I'm using

2.  Using agile principles to run operations - process techniques, like iterative development or even kanban/TPS kinds of process stuff.  Often with a goal of "faster!"

3.  Using automation - version control, automatic provisioning/control/monitoring.  Sometimes called "Infrastructure Automation" or similar.

This leads to some confusion, as most of these specific elements can be implemented in isolation.  For example, I think the discussion at OpsCamp about "Is DevOps an antipattern" was predicated on an assumption that DevOps meant only DevOps definition #2, "ops guys trying to be developers," and made the discussion somewhat odd to people with other assumed definitions.

I have a proposed set of definitions.  To explain it, let's look at Agile Development and see how it's defined.

Agile development, according to wikipedia and the agile manifesto, consists of a couple different "levels" of thing.  To sum up the wikipedia breakdown,

  • Agile Principles - like "business/users and developers working together."  These are the core values that inform agile, like collaboration, people over process, software over documentation, and responding to change over planning.
  • Agile Methods - specific process types.  Iterations, Lean, XP, Scrum.  "As opposed to waterfall."
  • Agile Practices - techniques often found in conjunction with agile development, not linked to a given method flavor, like test driven development, continuous integration, etc.

I believe the different parts of Agile Operations that people are talking about map directly to these three levels.

  • Agile Operations Principles includes things like dev/ops collaboration (DevOps definition 1 above); things like James Turnbull's 4-part model seem to be spot on examples of trying to define this arena.
  • Agile Operations Methods includes process you use to conduct operations - iterations, kanban, stuff you'd read in Visible Ops; Agile Operations definition #2 above.
  • Agile Operations Practices includes specific techniques like automated build/provisioning, monitoring, anything you'd have a "toolchain" for.  This contains DevOps definition #2 and Agile Operations definition #3 above.

I think it's helpful to break them up along the same lines as agile development, however, because in the end some of those levels should merge once developers understand ops is part of system development too...  There shouldn't be a separate "user/dev collaboration" and "dev/ops collaboration," in a properly mature model it should become a "user/dev/ops collaboration," for example.

I think the dev2ops guys' "People over Process over Tools" diagram mirrors this about exactly - the people being one of the important agile principles, process being a large part of the methods, and tools being used to empower the practices.

What I like about that diagram, and why I want to bring this all back to the Agile Manifesto discussion, is that the risk of having various sub-definitions increases the risk that people will implement the processes or tools without the principles in mind, which is definitely an antipattern.  The Agile guys would tell you that iterations without collaboration is likely to not work out real well.

And it happens in agile development too - there are some teams here at my company that have adopted the methods and/or tools of agile but not its principles, and the results are suboptimal.

Therefore I propose that "Agile Operations" is an umbrella term for all these things, and we keep in mind the principles/methods/practices differentiation.

If we want to call the principles "devops" for short and some of the practices "infrastructure automation" for short I think that would be fine...   Although dev/ops collaboration is ONE of the important principles - but probably not the entirety; and infrastructure automation is one of the important practices, but there are probably others.

11Mar/100

Upcoming Free Velocity WebOps Web Conference

O'Reilly's Velocity conference is the only generalized Web ops and performance conference out there.  We really like it; you can go to various other conferences and have 10-20% of the content useful to you as a Web Admin, or you can go here and have most of it be relevant!

They've been doing some interim freebie Web conferences and there's one coming up.  Check it out.  They'll be talking about performance functionality in Google Webmaster Tools, mySQL, Show Slow, provisioning tools, and dynaTrace's new AJAX performance analysis tool.

O'Reilly Velocity Online Conference: "Speed and Stability"
Thursday, March 17; 9:00am PST
Cost: Free

5Feb/104

OpsCamp Debrief

I went to OpsCamp this last weekend here in Austin, a get-togther for Web operations folks specifically focusing on the cloud, and it was a great time!  Here's my after action report.

The event invite said it was in the Spider House, a cool local coffee bar/normal bar.  I hadn't been there before, but other people that had said "That's insane!  They'll never fit that many people!  There's outside seating but it's freezing out!"  That gave me some degree of trepidation, but I still racked out in time to get downtown by 8 AM on a Saturday (sigh!).  Happily, it turned out that the event was really in the adjacent music/whatnot venue also owned by Spider House, the United States Art Authority, which they kindly allowed us to use for free!  There were a lot of people there; we weren't overfilling the place but it was definitely at capacity, there were near 100 people there.

I had just hears of OpsCamp through word of mouth, and figured it was just going to be a gathering of local Austin Web ops types.  Which would be entertaining enough, certainly.  But as I looked around the room I started recognizing a lot of guys from Velocity and other major shows; CEOs and other high ranked guys from various Web ops related tool companies.  Sponsors included John Willis and Adam Jacob (creator of Chef) from Opscode , Luke Kanies from Reductive Labs (creator of Puppet), Damon Edwards and Alex Honor from DTO Solutions (formerly ControlTier), Mark Hinkle and Matt Ray from Zenoss, Dave Nielsen (CloudCamp), Michael Coté (Redmonk), Bitnami, Spiceworks, and Rackspace Cloud.  Other than that, there were a lot of random Austinites and some guys from big local outfits (Dell, IBM).

You can read all the tweets about the event if you swing that way.

OpsCamp kinda grew out of an earlier thing, BarCampESM, also in Austin two years ago.  I never heard about that, wish I had.

How It Went

I had never been to an "unconference" before.  Basically there's no set agenda, it's self-emergent.  It worked pretty well.  I'll describe the process a bit for other noobs.

First, there was a round of lightning talks.  Brett from Rackspace noted that "size matters," Bill from Zenoss said "monitoring is important," and Luke from Reductive claimed that "in 2-4 years 'cloud' won't be a big deal, it'll just be how people are doing things - unless you're a jackass."

Then it was time for sessions.  People got up and wrote a proposed session name on a piece of paper and then went in front of the group and pitched it, a hand-count of "how many people find this interesting" was taken.

Candidates included:

  • service level to resolution
  • physical access to your cloud assets
  • autodiscovery of systems
  • decompose monitoring into tool chain
  • tool chain for automatic provisioning
  • monitoring from the cloud
  • monitoring in the cloud - widely dispersed components
  • agent based monitoring evolution
  • devops is the debil - change to the role of sysadmins
  • And more

We decided that so many of these touched on two major topics that we should do group discussions on them before going to sessions.  They were:

  • monitoring in the cloud
  • config mgmt in the cloud

This seemed like a good idea; these are indeed the two major areas of concern when trying to move to the cloud.

Sadly, the whole-group discussions, especially the monitoring one, were unfruitful.  For a long ass time people threw out brilliant quips about "Why would you bother monitoring a server anyway" and other such high-theory wonkery.  I got zero value out of these, which was sad because the topics were crucially interesting - just too unfocused; you had people coming at the problem 100 different ways in sound bytes.  The only note I bothered to write down was that "monitoring porn" (too many metrics) makes it hard to do correlation.  We had that problem here, and invested in a (horrors) non open-source tool, Opnet Panorama, that has an advanced analytics and correlation engine that can make some sense of tens of thousands of metrics for exactly that reason.

Sessions

There were three sessions.  I didn't take many notes in the first one because, being a Web ops guy, I was having to work a release simultaneously with attending OpsCamp 😛