Here at NI we did what I'll call a larval DevOps implementation starting about seven years ago when I came and took over our Web Systems team, essentially an applications administration/operations team for our Web site and other Web-related technologies.  There was zero automation and the model was very much "some developers show up with code and we had to put it in production and somehow deal with the many crashes per day resulting from it."  We would get 100-200 on-call pages a week from things going wrong in production.  We had especially entertaining weeks where Belgian hackers would replace pages on our site with French translations of the Hacker's Manifesto.  You know, standard Wild West stuff.  You've been there.

Step One: Partner With The Business

First thing I did (remember this is 2002), I partnered with the business leaders to get a "seat at the table" along with the development managers.  It turned out that our director of Web marketing was very open to the message of performance, availability, and security and gave us a lot of support.

This is an area where I think we're still ahead of even a lot of the DevOps message.  Agile development carries a huge tenet about developers partnering side-by-side with "the business" (end users, domain experts, and whatnot).  DevOps is now talking about Ops partnering with developers, but in reality that's a stab at the overall more successful model of "biz, dev, and ops all working together at once."

Of course we had ongoing relationships with the development managers, but it was obvious within a very short timeframe that wasn't going to be the end solution.  Sure, there was a little ossified opinion syndrome about the role of Ops (they are the blue collar "dirty people," as we all know) in relation to the developers - "You just move files, how hard is it?" - but mostly it was that our dev groups take a huge amount of very direct guidance and project management from our biz teams, and they would always be telling us "I don't have time to load test and my business analyst doesn't think it's important and says we have to release this week and I just do what they tell me."  The devs weren't empowered in and of themselves to do a lot of the stuff we needed them to do.  That made it critical to get direct business engagement.  Some environments (like my new group) have extremely empowered dev teams that are basically writing specs themselves; in that case probably starting with dev partnership would work well.

I'll also note that being aligned with the business helped us know what we should work on.  One year they'd be more interested in performance, another they'd be more interested in availability, etc.  That gave us the opportunity to be heroes - for example, when we switched over from our old load balancing software to our Citrix Netscalers and our global Web site performance improved by 50%.  We demonstrated "hey, some things Ops can give you, no programmers needed."  So then to a business guy you're just as valuable as a programmer.   They just want people to give them what they want, and if that's you then you're sitting pretty.

Step Two: Automate Some Stuff

Here's where I have to give huge credit to my partner in crime Peco.  I was making sure we were fixing root causes of problems and focusing on metrics to get us more people hired/permission to spend group time on improvement projects to ease the hundred pages a week problem.   Peco's a code hacker at heart so he quickly hit the two biggest pain points we needed to automate, Java application deployment and automated response to pages.  Our app deployment was manual up to that point - devs would hand over long incoherent install docs and we'd go perform them manually on multiple redundant servers.  He quickly wrote some scripts and an SSH framework and we made the developers structure their artifacts and fill out a config file that allowed us to take their apps from revision control and deploy them in an automated fashion to redundant dev, test, or production systems.  The devs loved it - we had a couple yap for the first week about "but but I like .wars not .ears" but the immense benefits became immediately apparent to them, largely thanks to our already pretty mature change control/release process.  Only ops could do deploys to test and we had big monthy Friday night releases for production code that devs had to attend, and dev time spent messing with dev deploys, waiting for test deploys, and staying up all night for production deploys plummeted.  A month later it was one of those "it's always been this way" steps.

After that we always wished we had time to go to a fuller automated CM solution, and as things like puppet and chef appeared we eyed them with interest, but we stemmed a huge part of the tide with that one Perl program.

Peco also wrote a framework to integrate with our monitoring solution and go do things like 'restart the server' without operator intervention.  Sure, there's down sides to that, but when you have people quitting your team because their oncall week involves restarting app servers all night every night, it serves a clear purpose.

These targeted automation steps moved us from chronic firefighting mode and pushed us over the hump to where we could actually try to make some more powerful changes.

Step Three: Value Added Services

Now that we had a handle on operations, we started to look for areas that no one was really addressing that needed help.  Performance/availability and security were clear weak spots and we had people with expertise and energy in both.  So we started what we called 'practice areas' and started in on tools and training in those areas. First we focused on detection and metric collection - showing the real performance of our Web site and individual apps and scanning for security holes.

The partnering with the business and dev managers was bearing fruit, in fact a Web steering project was established that had as its primary business goals for IT performance, availability, efficiency, TCO, and agility.  This created pressure on individual app teams to address the problems we could now detect, and we were ready with tools to help them fix it - for example, a shout out to Opnet's Panorama, an excellent application performance management (APM) tool that collects OS, software, and deep-dive Java metrics from across a distributed system and correlates those metrics.  We showed them "look, your app running in production isn't a mystery - you can get to it and look into it with this tool, and when there's a slowdown or failure you can get to where it is."  Similary, we implemented Splunk and gave developers constant access to their own log files.

From one point of view, it was us "passing the buck."  "Here, look at your own log files."  But the truth is, that the developers are the ones that wrote those cryptic error messages and know what they mean, and which errors are "normal" (to this day, whenever a developer tells me an error in their log file is "normal" I have to restrain the urge to punch them in the face).  By taking the tack of empowering devs to inspect their apps in production, in conjunction with producing pressure for them to improve operational aspects of their software,we were able to get the right people to work on the problems to really fix them.

Step Four: A Process Emerges

By this time the group had grown from the 2-3 people it started with to 7 people, including an operations guy in Hungary for follow the sun support.  In general we hired only high quality people; some were more experienced than others but we had high skills and esprit de corps.

There started to be a lot of pressure from production support duties interrupting project work.   Once we got large enough we split into a project subteam and an ops subteam to try to prevent that (what O'Reilly's Time Management for System Administrators book calls "interrupt shielding").  We did it as a "rotation" where admins would go onto "tech rotation" for 3 months and then back into projects, so that we could keep developing everyone and keep it "one team" and not "two teams" (no conceptual barrier causing finger pointing, communication hassle...)    We had talked about this when we were 4 people strong, and that wasn't enough people to make it tenable - and if we had grown to 10+ people I'd split it into two permanent groups, but for that middle area this model worked for us.

From the beginning, we had been advocating the idea of "you need to come and get an admin assigned to your project at kickoff, who will work with you to get all the systems stuff worked out."  This worked OK but was hit or miss - there was no formal project initiation process to hook into and so sometimes it happened, sometimes it didn't, sometimes it happened late.  But on projects where it happened, we had a proto-DevOps style ops person working with devs on a specific project.  Sadly though since there were so many projects, usually it wasn't embedded enough to work truly hand in hand (one ops person would be working on 5 projects at a time...).

Over time we did this enough that patterns began to emerge - what things need to happen in different kinds of engagements.  We then created a systems development process where each project needs to engage a Web Admin as part of their project, and it has a detailed playbook on what both the systems engineer and the developers need to do to make sure all the performance/availability/deployment/security/etc "nonfunctional requirements" are addressed.

Complications: Things That Didn't Work Well

Keep in mind we didn't have the full DevOps vision in mind when we started, we were evolving in the direction that seemed right to us as we got the opportunity over time.  So this all happened over years, whereas now I think with this blueprint in mind it could be implemented much more quickly even in an existing org.  These were the most significant issues that recurred while we were trying to do all this.

1. The Web Admins are a Web application administration group, but has to interact with other teams in our pretty large infrastructure group (UNIX admins, Windows admins, DBAs, Notes admins, network admins...  We tried to represent all the infrastructure groups to the programmers but, you know ops groups, there were constant outbreaks of squabbling.  A talk about this a bit in "Before DevOps, Don't You Need OpsOps?"  For a variety of reasons our relationships with the business (usually very good) and the devs (good and rocky alternating) were always better than our relationships with the other Infrastructure groups (always grumpy).
2. Our software dev process hasn't been very mature traditionally and has tended to scope projects down to single programmer size.  This has been changing over the last 2 years as they have embraced agile but there's still a tendency to have "many small projects."  As there were about 50 programmers and about 7 Web Admins (of which only 5 were available for projects), that caused a significant problem with trying to really partner with them on projects.  We tried focusing on "only large projects", but still you'd have projects where the ops person just didn't give the project enough time and attention.  And we uptook a lot of major complex platforms (Vignette VCM, Oracle SOA, etc.) that ended up leeching off a pretty constant amount of admin time to master and support (.5 person per platform in the steady state).  So the desire to do full DevOps partnering was always crushed under the wheels of a 1:10 ops:dev ratio.
3. It was also extremely harsh at a management level - our corporate culture, at least in IT, is very "relationship based" and not strongly management-driven.  I was used to "transactional based" organizations, where you go to a group and say "I need a something that is your group's job, so here's the request" and they do it.  Here you kinda have to go ask/sweet-talk/convince other groups to perform tasks.  Which is fine, it's an approach that other "consensus-driven" companies have scaled up with (HP, for example).  But in this role, we had to maintain relationships with a truly huge number of teams - a bunch of infrastructure teams, a bunch of Web programmer teams, and a bunch of Web business teams.  That level of serving/being a customer of 15 or so teams was just barely sustainable, but when we got tapped with running some internal systems like our company's SOA suite and internal collaboration system, and ten other customer and programming teams got dropped into the mix, it broke down (or at least that's where I reached the limit of my modest management skills).
   I think this is a place agile often hits rough patches - you bring in people from various disciplines to get the job done, but if the company's inherent organizational structure requires many separate org structures to do negotiation and resource management to make that work, you hit management paralysis (since the number of relationships is a factorial of the number of teams involved).  To scale you have to put some kind of lubricating process in place to enable that to happen, and we didn't have that.

Conclusion

We weren't operating in an agile process (iterations), but we were pretty successful at several important aspects of agile, especially those people usually mean when they say DevOps - partnering with business and developers, automating routine tasks, etc.  And we definitely build a large, skilled, and respected team that continually brought new tools and practices into the organization.  A lot of lessons learned that tell me that agile operations/DevOps is definitely the way to go, and now that we're getting to implement systems and processes from scratch (more on that later!) we have a lot of understanding of what we need to do/not do.

Killer quote:

"I say this as somebody who about 15 years ago chose system administration over development.  But system administration and system administrators have allowed themselves to lag in maturity behind what the state of the art is. These new technologies are finally causing us to be held to account to modernize the way we do things.  And I think that’s a welcome and healthy challenge."

Some customers have asked us about ways to easily convert virtual machines from VMware vSphere, Citrix Xen Server, and Microsoft Hyper-V to Amazon EC2 instances - and vice versa. If this is something that you're interested in, we would like to hear from you. Please send an email to aws-vm@amazon.com describing your needs and use case.

I'll share my reply here for comment!

This is a killer feature that allows a number of important activities.

1.  Product VMs.  Many suppliers are starting to provide third-party products in the form of VMs instead of software to ease install complexity, or in an attempt to move from a hardware appliance approach to a more-software approach.  This pretty much prevents their use in EC2.  <cue sad music>  As opposed to "Hey, if you can VM-ize your stuff then you're pretty close to being able to offer it as an Amazon AMI or even SaaS offering."  <schwing!>

2.  Leveraging VM Investments.  For any organization that already has a VM infrastructure, it allows for reduction of cost and complexity to be able to manage images in the same way.  It also allows for the much promised but under-delivered "cloud bursting" theory where you can run the same systems locally and use Amazon for excess capacity.  In the current scheme I could make some AMIs "mostly" like my local VMs - but "close" is not good enough to use in production.

3.  Local testing.  I'd love to be able to bring my AMIs "down to me" for rapid redeploy.  I often find myself having to transfer 2.5 gigs of software up to the cloud, install it, find a problem, have our devs fix it and cut another release, transfer it up again (2 hour wait time again, plus paying $$ for the transfer)...

4.  Local troubleshooting. We get an app installed up in the cloud and it's not acting quite right and we need to instrument it somehow to debug.  This process is much easier on a local LAN with the developers' PCs with all their stuff installed.

5.  Local development. A lot of our development exercises the Amazon APIs.  This is one area where Azure has a distinct advantage and can be a threat; in Visual Studio there is a "local Azure fabric" and a dev can write their app and have it running "in Azure" but on their machine, and then when they're ready deploy it up.  This is slightly more than VM consumption, it's VMs plus Eucalyptus or similar porting of the Amazon API to the client side, but it's a killer feature.

Xen or VMWare would be fine - frankly this would be big enough for us I'd change virtualization solutions to the one that worked with EC2.

I just asked one of our developers for his take on value for being able to transition between VMs and EC2 to include in this email, and his response is "Well, it's just a no-brainer, right?"  Right.

I missed the first one but loved OpsCamp so I'm going!

When  you start an "instance-store" Amazon EC2 instance, you get a certain amount of ephemeral storage allocated and mounted automatically.  The amount of space varies by instance size and is defined here.  The storage location and format also varies by instance size and is defined here.

The upshot is that if you start an "instance-store" small Linux EC2 instance, it automagically has a free 150 GB /mnt disk and a 1 GB swap partition up and runnin' for ya.  (mount points vary by image, but that's where they are in the Amazon Fedora starter.)

[root@domU-12-31-39-00-B2-01 ~]# df -k

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   1636668   8160252  17% /
/dev/sda2            153899044    192072 145889348   1% /mnt
none                    873828         0    873828   0% /dev/shm

[root@domU-12-31-39-00-B2-01 ~]# free
total       used       free     shared    buffers     cached
Mem:       1747660      84560    1663100          0       4552      37356
-/+ buffers/cache:      42652    1705008
Swap:       917496          0     917496

But, you say, I am not old or insane!  I use EBS-backed images, just as God intended.  Well, that's a good point.  But when you pull up an EBS image, these ephemeral disk areas are not available to you.  The good news is, that's just by default.

The ephemeral storage is still available and can be used (for free!) by an EBS-backed image.  You just have to set the block devices up either explicitly when you run the instance or bake them into the image.

Runtime:

You refer to the ephemeral chunks as "ephemeral0", "ephemeral1", etc. - they don't tell you explicitly which is which but basically you just count up based on your instance type (review the doc).  For a small image, it has an ephemeral0 (ext3, 15 GB) and an ephemeral1 (swap, 1 GB).  To add them to an EBS instance and mount them in the "normal" places, you do:

ec2-run-instances <ami id> -k <your key> --block-device-mapping '/dev/sda2=ephemeral0'
--block-device-mapping '/dev/sda3=ephemeral1'

On the instance you have to mount them - add these to /etc/fstab and mount -a or do whatever else it is you like to do:

/dev/sda3                 swap                    swap    defaults 0 0
/dev/sda2                 /mnt                    ext3    defaults 0 0

And if you want to turn the swap on immediately, "swapon /dev/sda3".

Image:

You can also bake them into an image.  Add a fstab like the one above and when you create the image, do it like this, using the exact same --block-device-mapping flag:

ec2-register -n <ami id> -d "AMI Description" --block-device-mapping  /dev/sda2=ephemeral0
--block-device-mapping '/dev/sda3=ephemeral1' --snapshot your-snapname --architecture i386
--kernel<aki id>  --ramdisk <ari id>

Ta da. Free storage that doesn't persist.  Very useful as /tmp space.  Opinion is split among the Linuxerati about whether you want swap space nowadays or not; some people say some mix of  "if you're using more than 1.8 GB of RAM you're doing it wrong" and "swapping is horrid, just let bad procs die due to lack of memory and fix them."  YMMV.

Ephemeral EBS?

As another helpful tip, let's say you're adding an EBS to an image that you don't want to be persistent when the instance dies.  By default, all EBSes are persistent and stick around muddying up your account till you clean them up.   If you don't want certain EBS-backed drives to persist, what you do is of the form:

ec2-modify-instance-attribute --block-device-mapping "/dev/sdb=vol-f64c8e9f:true" i-e2a0b08a

Where 'true' means "yes, please, delete me when I'm done."  This command throws a stack trace to the tune of

Unexpected error: java.lang.ClassCastException: com.amazon.aes.webservices.client.InstanceBlockDeviceMappingDescription
cannot be cast to com.amazon.aes.webservices.client.InstanceBlockDeviceMappingResponseDescription

But it works, that's just a lame API tools bug.

If you're from the Austin area you probably know National Instruments - our campus is up here at Mopac and Braker; we've been named one of Fortune Magazine's 100 Best Companies To Work For eleven years running.

We've got an opening on a new team building cloud-based Web systems for new SaaS products and Web integration features of our existing hardware and software products.

We need someone to form the core of a new international Web Operations team to provide 24x7 support of these products. You'll work interactively with our R&D engineers, Web programmers, systems engineers, product support organization, and a host of other groups to this end.  You'll also help build up the operational environment - monitoring, provisioning, release process, high availability, reporting, performance assurance, security auditing, that kind of thing.  You’ll need to be able to document well and train other operations staff.

You can read the "straight" job posting and apply at ni.com/jobs under R&D as "Web Systems Engineer."

Benefits:

• Work with smart people
• Fun, fast-paced environment
• We'll pay you in real money, not WoW gold
• We're growing and profitable ::cough no layoffs cough::
• Room to innovate

You should be skilled in administration of Linux and Windows systems (we run about 50/50), Java and .NET app servers, Web services, search, security, performance, high availability, infrastructure automation, and all the other fun things that Web systems people do.

This is a Web systems administration position with responsibility for realtime operations - it's not Web design, Web applications programming, help desk/end user support, or "back room" systems admin. If you don't want to spend your day configuring load balancing software, figuring out why someone's app is crashing the app server, finding security holes in a Web application, debugging why a new Web page is really slow, getting pages from server monitors, and coding up a great new tool to automate your team's work, this isn't the position for you.  There is oncall “pager duty” and some nights/weekends required.

Sorry, no recruiters or H1-B candidates.

• Internal Borders by Graham Bleach
• DevOps is a good cause, but what about OpsOps? by the Build Doctor
• DevOps, SecOps, DBAOps, NetOps by Kris Buytaert

This resonates deeply with me.  I've seen that problem in spades.  I think in general that a lot of the discussion about the agile ops space is too simplistic in that it seems tuned to organizations of "five guys, three of whom are coders and two of whom are operations" and there's no differentiation.  In real life, there's often larger orgs and a lot of differentiation that causes various collaboration challenges.  Some people refer to this as Web vs Enterprise, but I don't think that's strictly true; once your Web shop grows from 5 guys to 200 it runs afoul of this too - it's a simple scalability and organizational engineering problem.

As an aside, I don't even like the "Ops" term - a sysadmin team can split into subgroups that do systems engineering, release management, and operational support...  Just saying "Ops" seems to me to create implications of not being a partner in the initial design and development of the overall system/app/service/site/whatever you want to call it.

Ops Verticals

Here, we have a large Infrastructure department.  Originally, it was completely siloed by technology verticals, and there's a lot of subgroups.  Network, UNIX, Windows, DBA, Lotus Notes, Telecom, Storage, Data Center...  Some ten plus years ago when the company launched their Web site in earnest, they quickly realized that wasn't going to work out.  You had the buck-passing behavior described in the blog posts above that made issues impossible to solve in a timely fashion, plus it made collaboration with devs/business nearly impossible.  Not only did you need like 8 admins to come involve themselves in your project, but they did not speak similar enough languages - you'd have some crusty UNIX admin yelling "WHAT ABOUT THE INODES" until the business analyst started to cry.

Dev Silos

But are our developers here better off?  They are siloed by business unit.  Just among the Web developers there's the eCommerce developers, eCRM, Product Advisors, Community, Support, Content Management...  On the one hand, they are able to be very agile in creating solutions inside their specific niche.  On the other hand, they are all working within the same system environment, and they don't always stay on the same page in terms of what technologies they are using. "Well, I'm sure THAT team bought a lovely million dollar CMS, but we're going to buy our own different million dollar CMS.   No, you don't get more admin resource."  Over time, they tried to produce architecture groups and other cross-team initiatives to try to rein in the craziness, with mixed but overall positive results.

Plugging the Dike

What we did was create a Web Administration group (Web Ops, whatever you want to call it) that was holistically responsible for Web site uptime, performance, and security.  Running that team was my previous gig, did it for five years.  That group was more horizontally focused and would serve as an interface to the various technology verticals; it worked closely with developers in system design during development, coordinated the release process, and involved devs in troubleshooting during the production phase.

BizOps?

In fact, we didn't just partner with the developers - we partnered with the business owners of our Web site too, instead of tolerating the old model of "Business collaborates with the developers, who then come and tell ops what to do."  This was a remarkably easy sell really.  The company lost money every minute the Web site was down, and it was clear that the dev silos weren't going to be able to fix that any more than the ops silos were.  So we quickly got a seat at the same table.

Results

This was a huge success.  To this day, our director of Web Marketing is one of the biggest advocates of the Web operations team.  Since then, other application administration (our word for this cross-disciplinary ops) teams have formed along the same model.  The DevOps collaboration has been good overall - with certain stresses coming from the Web Ops team's role as gatekeeper and process enforcement.  Ironically, the biggest issues and worst relationships were within Infrastructure between the ops teams!

OpsOps - The Fly In The Ointment

The ops team silos haven't gone down quietly.  To this day the head DBA still says "I don't see a good reason for you guys [WebOps] to exist."  I think there's a common "a thing is just the sum of its parts" mindset among admins for whatever reason.  There are also turf wars arising from the technology silo division and the blurring of technology lines by modern tech.  I tried again and again to pitch "collaborative system administration."  But the default sysadmin behavior is to say "these systems are mine and I have root on them.  Those are your systems and you have root on them.  Stay on your side of the line and I'll stay on mine."

Fun specific Catch-22 situations we found ourselves in:

• Buying a monitoring tool that correlates events across all the different tiers to help root-cause production problems - but the DBAs refusing to allow it on "their" databases.
• Buying a hardware load balancer - we were going to manage it, not the network team, and it wasn't a UNIX or Windows server, so we couldn't get anyone to rack and jack it (and of course we weren't allowed to because "Why would a webops person need server room access, that's what the other teams are for").

Some of the problem is just attitude, pure and simple.  We had problems even with collaboration inside the various ops teams!  We'd work with one DBA to design a system and then later need to get support from another DBA, who would gripe that "no one told/consulted them!"  Part of the value of the agile principles that "DevOps" tries to distill is just a generic "get it into your damn head you need to be communicating and working together and that needs to be your default mode of operation." I think it's great to harp on that message because it's little understood among ops.  For every dev group that deliberately ostracizes their ops team, there's two ops teams who don't think they need to talk to the devs - in the end, it's mostly our fault.

Part of the problem is organizational.  I also believe (and ITIL, I think, agrees with me) that the technology-silo model has outlived its usefulness.  I'd like to see admin teams organized by service area with integral DBAs, OS admins, etc.  But people are scared of this for a couple reasons.  One is that those admins might do things differently from area to area (the same problem we have with our devs) - this could be mitigated by "same tech" cross-org standards/discussions.  The other is that this model is not the cheapest.  You can squeeze every last penny out if you only have 4 Windows admins and they're shared by 8 functional areas.  Of course, you are cutting off your nose to spite your face because you lose lots more in abandoned agility, but frankly corporate finance rules (minimize G&A spending) are a powerful driver here.

If nothing else, there's not "one right organization" - I'd be tempted to reorg everyone from verticals into horizontals, let that run for 5 years, and then reorg back the other way, just to keep the stratification from setting in.

Specialist vs Generalist

One other issue.  The Web Ops team we created required us to hire generalists - but generalists that knew their stuff in a lot of different areas.  It became very hard to hire for that position and training took months before someone was at all effective.  Being a generalist doesn't scale well.  Specialization is inevitable and, indeed, desirable (as I think pretty much anything in the history of anything demonstrates).  You can mitigate that with some cross-training and having people be generalists in some areas, but in the end, once you get past that "three devs, two ops, that's the company" model, specialization is needed.

That's why I think one of the common definitions of DevOps - all ops folks learning to be developers or vice versa - is fundamentally flawed.  It's not sustainable.  You either need to hire all expensive superstars that can be good at both, or you hire people that suck at both.

What you do is have people with varying mixes.  In my current team we have a continuum of pure ops people, ops folks doing light dev, devs doing light ops, and pure devs.  It's good to have some folks who are generalizing and some who are specializing.  It's not specializing that is bad, it's specialists who don't collaborate that are bad.

Conclusion

So I've shared a lot of experiences and opinions above but I'm not sure I have a brilliant solution to the problem.  I do think we need to recognize that Ops/Ops collaboration is an issue that arises with scale and one potentially even harder to overcome than Dev/Ops collaboration.  I do think stressing collaboration as a value and trying to break down organizational silos may help.  I'd be happy to hear other folks' experiences and thoughts!

The first is DevOps, which has two totally different frequently used definitions.

1.  Developers and Ops working closely together - the "hugs and collaboration" definition

2.  Operations folks uptaking development best practices and writing code for system automation

The second is Agile Operations, which also has different meanings.

1.  Same as DevOps, whichever definition of that I'm using

2.  Using agile principles to run operations - process techniques, like iterative development or even kanban/TPS kinds of process stuff.  Often with a goal of "faster!"

3.  Using automation - version control, automatic provisioning/control/monitoring.  Sometimes called "Infrastructure Automation" or similar.

This leads to some confusion, as most of these specific elements can be implemented in isolation.  For example, I think the discussion at OpsCamp about "Is DevOps an antipattern" was predicated on an assumption that DevOps meant only DevOps definition #2, "ops guys trying to be developers," and made the discussion somewhat odd to people with other assumed definitions.

I have a proposed set of definitions.  To explain it, let's look at Agile Development and see how it's defined.

Agile development, according to wikipedia and the agile manifesto, consists of a couple different "levels" of thing.  To sum up the wikipedia breakdown,

• Agile Principles - like "business/users and developers working together."  These are the core values that inform agile, like collaboration, people over process, software over documentation, and responding to change over planning.
• Agile Methods - specific process types.  Iterations, Lean, XP, Scrum.  "As opposed to waterfall."
• Agile Practices - techniques often found in conjunction with agile development, not linked to a given method flavor, like test driven development, continuous integration, etc.

I believe the different parts of Agile Operations that people are talking about map directly to these three levels.

• Agile Operations Principles includes things like dev/ops collaboration (DevOps definition 1 above); things like James Turnbull's 4-part model seem to be spot on examples of trying to define this arena.
• Agile Operations Methods includes process you use to conduct operations - iterations, kanban, stuff you'd read in Visible Ops; Agile Operations definition #2 above.
• Agile Operations Practices includes specific techniques like automated build/provisioning, monitoring, anything you'd have a "toolchain" for.  This contains DevOps definition #2 and Agile Operations definition #3 above.

I think it's helpful to break them up along the same lines as agile development, however, because in the end some of those levels should merge once developers understand ops is part of system development too...  There shouldn't be a separate "user/dev collaboration" and "dev/ops collaboration," in a properly mature model it should become a "user/dev/ops collaboration," for example.

I think the dev2ops guys' "People over Process over Tools" diagram mirrors this about exactly - the people being one of the important agile principles, process being a large part of the methods, and tools being used to empower the practices.

What I like about that diagram, and why I want to bring this all back to the Agile Manifesto discussion, is that the risk of having various sub-definitions increases the risk that people will implement the processes or tools without the principles in mind, which is definitely an antipattern.  The Agile guys would tell you that iterations without collaboration is likely to not work out real well.

And it happens in agile development too - there are some teams here at my company that have adopted the methods and/or tools of agile but not its principles, and the results are suboptimal.

Therefore I propose that "Agile Operations" is an umbrella term for all these things, and we keep in mind the principles/methods/practices differentiation.

If we want to call the principles "devops" for short and some of the practices "infrastructure automation" for short I think that would be fine...   Although dev/ops collaboration is ONE of the important principles - but probably not the entirety; and infrastructure automation is one of the important practices, but there are probably others.

They've been doing some interim freebie Web conferences and there's one coming up.  Check it out.  They'll be talking about performance functionality in Google Webmaster Tools, mySQL, Show Slow, provisioning tools, and dynaTrace's new AJAX performance analysis tool.

O'Reilly Velocity Online Conference: "Speed and Stability"
Thursday, March 17; 9:00am PST
Cost: Free