Here at NI we did what I'll call a larval DevOps implementation starting about seven years ago when I came and took over our Web Systems team, essentially an applications administration/operations team for our Web site and other Web-related technologies.  There was zero automation and the model was very much "some developers show up with code and we had to put it in production and somehow deal with the many crashes per day resulting from it."  We would get 100-200 on-call pages a week from things going wrong in production.  We had especially entertaining weeks where Belgian hackers would replace pages on our site with French translations of the Hacker's Manifesto.  You know, standard Wild West stuff.  You've been there.

Step One: Partner With The Business

First thing I did (remember this is 2002), I partnered with the business leaders to get a "seat at the table" along with the development managers.  It turned out that our director of Web marketing was very open to the message of performance, availability, and security and gave us a lot of support.

This is an area where I think we're still ahead of even a lot of the DevOps message.  Agile development carries a huge tenet about developers partnering side-by-side with "the business" (end users, domain experts, and whatnot).  DevOps is now talking about Ops partnering with developers, but in reality that's a stab at the overall more successful model of "biz, dev, and ops all working together at once."

Of course we had ongoing relationships with the development managers, but it was obvious within a very short timeframe that wasn't going to be the end solution.  Sure, there was a little ossified opinion syndrome about the role of Ops (they are the blue collar "dirty people," as we all know) in relation to the developers - "You just move files, how hard is it?" - but mostly it was that our dev groups take a huge amount of very direct guidance and project management from our biz teams, and they would always be telling us "I don't have time to load test and my business analyst doesn't think it's important and says we have to release this week and I just do what they tell me."  The devs weren't empowered in and of themselves to do a lot of the stuff we needed them to do.  That made it critical to get direct business engagement.  Some environments (like my new group) have extremely empowered dev teams that are basically writing specs themselves; in that case probably starting with dev partnership would work well.

I'll also note that being aligned with the business helped us know what we should work on.  One year they'd be more interested in performance, another they'd be more interested in availability, etc.  That gave us the opportunity to be heroes - for example, when we switched over from our old load balancing software to our Citrix Netscalers and our global Web site performance improved by 50%.  We demonstrated "hey, some things Ops can give you, no programmers needed."  So then to a business guy you're just as valuable as a programmer.   They just want people to give them what they want, and if that's you then you're sitting pretty.

Step Two: Automate Some Stuff

Here's where I have to give huge credit to my partner in crime Peco.  I was making sure we were fixing root causes of problems and focusing on metrics to get us more people hired/permission to spend group time on improvement projects to ease the hundred pages a week problem.   Peco's a code hacker at heart so he quickly hit the two biggest pain points we needed to automate, Java application deployment and automated response to pages.  Our app deployment was manual up to that point - devs would hand over long incoherent install docs and we'd go perform them manually on multiple redundant servers.  He quickly wrote some scripts and an SSH framework and we made the developers structure their artifacts and fill out a config file that allowed us to take their apps from revision control and deploy them in an automated fashion to redundant dev, test, or production systems.  The devs loved it - we had a couple yap for the first week about "but but I like .wars not .ears" but the immense benefits became immediately apparent to them, largely thanks to our already pretty mature change control/release process.  Only ops could do deploys to test and we had big monthy Friday night releases for production code that devs had to attend, and dev time spent messing with dev deploys, waiting for test deploys, and staying up all night for production deploys plummeted.  A month later it was one of those "it's always been this way" steps.

After that we always wished we had time to go to a fuller automated CM solution, and as things like puppet and chef appeared we eyed them with interest, but we stemmed a huge part of the tide with that one Perl program.

Peco also wrote a framework to integrate with our monitoring solution and go do things like 'restart the server' without operator intervention.  Sure, there's down sides to that, but when you have people quitting your team because their oncall week involves restarting app servers all night every night, it serves a clear purpose.

These targeted automation steps moved us from chronic firefighting mode and pushed us over the hump to where we could actually try to make some more powerful changes.

Step Three: Value Added Services

Now that we had a handle on operations, we started to look for areas that no one was really addressing that needed help.  Performance/availability and security were clear weak spots and we had people with expertise and energy in both.  So we started what we called 'practice areas' and started in on tools and training in those areas. First we focused on detection and metric collection - showing the real performance of our Web site and individual apps and scanning for security holes.

The partnering with the business and dev managers was bearing fruit, in fact a Web steering project was established that had as its primary business goals for IT performance, availability, efficiency, TCO, and agility.  This created pressure on individual app teams to address the problems we could now detect, and we were ready with tools to help them fix it - for example, a shout out to Opnet's Panorama, an excellent application performance management (APM) tool that collects OS, software, and deep-dive Java metrics from across a distributed system and correlates those metrics.  We showed them "look, your app running in production isn't a mystery - you can get to it and look into it with this tool, and when there's a slowdown or failure you can get to where it is."  Similary, we implemented Splunk and gave developers constant access to their own log files.

From one point of view, it was us "passing the buck."  "Here, look at your own log files."  But the truth is, that the developers are the ones that wrote those cryptic error messages and know what they mean, and which errors are "normal" (to this day, whenever a developer tells me an error in their log file is "normal" I have to restrain the urge to punch them in the face).  By taking the tack of empowering devs to inspect their apps in production, in conjunction with producing pressure for them to improve operational aspects of their software,we were able to get the right people to work on the problems to really fix them.

Step Four: A Process Emerges

By this time the group had grown from the 2-3 people it started with to 7 people, including an operations guy in Hungary for follow the sun support.  In general we hired only high quality people; some were more experienced than others but we had high skills and esprit de corps.

There started to be a lot of pressure from production support duties interrupting project work.   Once we got large enough we split into a project subteam and an ops subteam to try to prevent that (what O'Reilly's Time Management for System Administrators book calls "interrupt shielding").  We did it as a "rotation" where admins would go onto "tech rotation" for 3 months and then back into projects, so that we could keep developing everyone and keep it "one team" and not "two teams" (no conceptual barrier causing finger pointing, communication hassle...)    We had talked about this when we were 4 people strong, and that wasn't enough people to make it tenable - and if we had grown to 10+ people I'd split it into two permanent groups, but for that middle area this model worked for us.

From the beginning, we had been advocating the idea of "you need to come and get an admin assigned to your project at kickoff, who will work with you to get all the systems stuff worked out."  This worked OK but was hit or miss - there was no formal project initiation process to hook into and so sometimes it happened, sometimes it didn't, sometimes it happened late.  But on projects where it happened, we had a proto-DevOps style ops person working with devs on a specific project.  Sadly though since there were so many projects, usually it wasn't embedded enough to work truly hand in hand (one ops person would be working on 5 projects at a time...).

Over time we did this enough that patterns began to emerge - what things need to happen in different kinds of engagements.  We then created a systems development process where each project needs to engage a Web Admin as part of their project, and it has a detailed playbook on what both the systems engineer and the developers need to do to make sure all the performance/availability/deployment/security/etc "nonfunctional requirements" are addressed.

Complications: Things That Didn't Work Well

Keep in mind we didn't have the full DevOps vision in mind when we started, we were evolving in the direction that seemed right to us as we got the opportunity over time.  So this all happened over years, whereas now I think with this blueprint in mind it could be implemented much more quickly even in an existing org.  These were the most significant issues that recurred while we were trying to do all this.

1. The Web Admins are a Web application administration group, but has to interact with other teams in our pretty large infrastructure group (UNIX admins, Windows admins, DBAs, Notes admins, network admins...  We tried to represent all the infrastructure groups to the programmers but, you know ops groups, there were constant outbreaks of squabbling.  A talk about this a bit in "Before DevOps, Don't You Need OpsOps?"  For a variety of reasons our relationships with the business (usually very good) and the devs (good and rocky alternating) were always better than our relationships with the other Infrastructure groups (always grumpy).
2. Our software dev process hasn't been very mature traditionally and has tended to scope projects down to single programmer size.  This has been changing over the last 2 years as they have embraced agile but there's still a tendency to have "many small projects."  As there were about 50 programmers and about 7 Web Admins (of which only 5 were available for projects), that caused a significant problem with trying to really partner with them on projects.  We tried focusing on "only large projects", but still you'd have projects where the ops person just didn't give the project enough time and attention.  And we uptook a lot of major complex platforms (Vignette VCM, Oracle SOA, etc.) that ended up leeching off a pretty constant amount of admin time to master and support (.5 person per platform in the steady state).  So the desire to do full DevOps partnering was always crushed under the wheels of a 1:10 ops:dev ratio.
3. It was also extremely harsh at a management level - our corporate culture, at least in IT, is very "relationship based" and not strongly management-driven.  I was used to "transactional based" organizations, where you go to a group and say "I need a something that is your group's job, so here's the request" and they do it.  Here you kinda have to go ask/sweet-talk/convince other groups to perform tasks.  Which is fine, it's an approach that other "consensus-driven" companies have scaled up with (HP, for example).  But in this role, we had to maintain relationships with a truly huge number of teams - a bunch of infrastructure teams, a bunch of Web programmer teams, and a bunch of Web business teams.  That level of serving/being a customer of 15 or so teams was just barely sustainable, but when we got tapped with running some internal systems like our company's SOA suite and internal collaboration system, and ten other customer and programming teams got dropped into the mix, it broke down (or at least that's where I reached the limit of my modest management skills).
   I think this is a place agile often hits rough patches - you bring in people from various disciplines to get the job done, but if the company's inherent organizational structure requires many separate org structures to do negotiation and resource management to make that work, you hit management paralysis (since the number of relationships is a factorial of the number of teams involved).  To scale you have to put some kind of lubricating process in place to enable that to happen, and we didn't have that.

Conclusion

We weren't operating in an agile process (iterations), but we were pretty successful at several important aspects of agile, especially those people usually mean when they say DevOps - partnering with business and developers, automating routine tasks, etc.  And we definitely build a large, skilled, and respected team that continually brought new tools and practices into the organization.  A lot of lessons learned that tell me that agile operations/DevOps is definitely the way to go, and now that we're getting to implement systems and processes from scratch (more on that later!) we have a lot of understanding of what we need to do/not do.

The book is about technical recruiting and retention, and even though it’s a small format under 200 page book, it covers a lot of different topics.  His focus is on hiring programmers but I think a lot of the same principles apply to hiring for systems admin/Web systems positions.  Hiring has been one of the hardest parts of being a Web systems manager, so I got a lot out of the book and tried putting it into practice.  Results detailed below!

The Book

The first chapter talks about the relative effectiveness of programmers.  We often hire programmers and pay the good ones 10% more than the bad ones.  But he has actual data, drawn from a Yale professor who repeatedly teaches the same CS class and assigns the same projects, which shows something that those of us who have been in the field for a long time know – which is that the gap in achievement between the best programmers and the worst ones is a factor of ten.  That’s right.  In a highly controlled environment, the best programmers completed projects 3-4 times faster than the average and 10x faster than the slowest ones.  (And this same relationship holds when adjusting for quality of results.)  I’ve been in IT for 15 years and I can guarantee this is true.  You can give the same programming task to a bunch of different programmers and get results from “Here, I did it last night” to “Oh, that’ll take three months.”  He goes on to note other ways in which you can get 10 mediocre programmers that cannot achieve the same “high notes” as one good programmer.  This goes to reinforce how important the programmer, as human capital, is to an organization.

Next, he delves into how you find good developers.  Unfortunately, the easy answers don’t work.  Posting on monster.com or craigslist gets lots of hits but few keeps.  Employee referrals don’t always get the best people either.  How do you?  He has three suggestions.

1. Go to the mountain
2. Internships
3. Build your own community

“Go to the mountain” means to figure out where the smart people are that you want to hire, and go hang out there.  Conferences.  Organizations.  Web sites.  General job sites are zoos, you need venues that are more specifically spot on.  Want a security guy?  Post on OWASP or ISSA forums, not monster.

We do pretty well with internships, even enhancing that with company sponsored student sourcing/class projects and a large campus recruiting program.  He has some good sub-points however – like make your offers early.  If you liked them as an intern, offer them a full-time job at that point for when they graduate, don't wait.  Waiting puts you into more of a competitive situation.  And interns should be paid, given great work to do, and courted for the perm job during the internship.

Building a community – he acknowledges that’s hard.  Our company has external communities but not really for IT.  For a lot of positions we should be on our our forums like fricking scavengers trying to hire people that post there.

Chapter 3 is interesting; it’s a “Field Guide to Developers.”  It seems obvious, but to get the best developers working for you, you need to be showing that you’re the kind of place they want to work.  What does the wily developer like?

First, there’s the physical workspace.  Does it look cool or awful?  You have to turn around and take a look at yourself as an outsider would.  A big monochrome cube farm, with crappy chairs and computers, and a lobby with wilting plants in it is a warning sign to developers to stay away.   Private offices with Aeron chairs and dual monitors say “no really, we do consider our people more than interchangeable parts.”  You can consider these kinds of things luxuries, but good programmers can go a hundred other places that have them.  We only bat about .500 on this, and that’s mostly due to our beautiful campus.

Side note - we had one guy interview here, who when asked what he found important about a job, replied "a comfy chair and nice monitor."  It's OK for that to be important to you but when it's first thing out in an interview, it makes you seem like you are an entitlement fan and don't really care about your work - we passed on that guy.  Right?  Wrong?  Chime in with a comment.

Next, there’s the interpersonal environment.  Are programmers treated well in the organization, or are they typists given orders from on high?  What are their colleagues like – when they meet people during the interviews, they are always asking themselves “do I want to work with this person?”  Grumpy, bedraggled, unprepared, or distracted says “no.”  He notes at Fog Creek they started with the old Microsoft hiring maxim of “Smart, and gets things done” but soon learned that they (unlike Microsoft) needed to add a third clause, “Not a jerk.”  Does it look like they’ll be given responsibility and autonomy?  Is the environment rife with politics?  Will they be working with cool new technology, on something interesting?   Can they identify with the company or otherwise be attracted by its goals?  Even “side” goals like a wellness program or recycling drive count here.  For anyone who hasn’t noticed, this is the “me” generation.  Will I like working there, will it be happy and beneficial for me day in, day out?

He also notes another truism that too many people don’t believe, which is what programmers don’t care about – money.  (Within reason of course.)  It’s very low on the list and usually only comes up when the other factors are seen as undesirable.

“Sorting Resumes” is the topic of Chapter 4.  He acknowledges how terrible a tool the average resume and cover letter are in terms of conveying anything you really want to know about a candidate.  You can’t hire from them, you can only screen out people you clearly don’t want to hire from them.

Joel’s criteria for sorting resumes are:

• Passion – evidence that they love programming, based on a variety of factors – into computers from an early age, extracurricular computer activities (like working on open source projects), interest in new technologies.
• Pickiness – I call this “fit” - is the applicant looking to work for you?  Did they bother to customize the cover letter?  Remember it’s not all about whether you like them; if you extend an offer you want to have a high probability that they’ll accept.
• English – if your communication skills are so poor that you can’t even get a resume right, when you have an infinite amount of time to craft it and get other people to proofread it, you’re trouble.
• Brains – anything indicating they’re just flat smart.
• Selectivity – if they went through a highly selective process in the past (getting into an Ivy League school, hired by Google) then someone else has pre-vetted them for you.
• Hard-core – certain technologies are harder than others.  If someone’s done C++ programming, they are in general going to have more chops than someone who’s only programmed JavaScript, even if you’re looking for a JavaScript programmer.
• Diversity – not gratuitous diversity, but you want to make sure and have different types of people on your teams.  If your team is all old fogeys, hire someone right out of school so that you can benefit from a different viewpoint.

He also talks about what not to do.  He’s all about making them show some programming chops during an interview, but putting an additional hoop of a technical test into the prequalification process just loses you people who think that’s lame and they can get a good job without being your trained monkey.  He also notes it’s important not to look too much for experience with specific technologies.  The “keyword matching” school of programmer selection is awful.  A good programmer that knows 3 languages can pick up whatever language number 4 you use without any problem, whereas some goon who only knows language 4 will, after about two weeks, be less productive than the other guy forever.  Every once in a while you are specifically looking for an expert, in which case that’s fine – if you want a Java architect for the department then you do want in depth expertise in it.

Chapter 4 covers the phone screen.  He recommends starting with a phone screen, because it actually allows you to do a first cut more “fairly” without seeing the candidate.  And it’s less expensive than an in-person.  His phone interviews have three parts.  The first is the ”tell me about yourself” part.  In this he listens for signs of passion and being a problem solver, and drills down into technology and politics.  By technology, he means establishing exactly what role they played with the technologies on their resume and in their story, to find their real level of technical skill.  And by politics, he really means drive and persistence – how they handle challenges and get things done.

Next, he poses an open-ended technical question to get in tune with their thought processes.  Not looking for pseudocode as an output, but for their ability to make tradeoffs, understand subtleties, etc.  He gives examples like “How would you design a program that lets people play Monopoly with each other over the Internet?”   Something they’d be familiar with but not have implemented.  The key premise here is that smart people can tell other smart people by having an in depth discussion with them over a problem at hand.  Again, having the same problem posed to all candidates allows for quality compare/contrast.

The third and final part of his phone screens are letting the candidate interview him, to aggressively sell them on the position and also to see if they’ve done any basic research into the company, which reveals how serious they are.  Remember, it’s all about closing the deal, so whether you want them is only half of the equation ; are they seriously looking as well and do they want you.

Ernest’s Conclusion: This book is the God’s honest truth and you would do well to read it and follow it.  Sure, he has a couple specific hang-ups that can safely be ignored – how cool iPods are, how important private offices are, and that programmers should fly first class – but every core idea here is sound and proven over and over again in practice.

Let's Use What We've Learned

So let’s go through his process as a test case.  I need a Web Systems engineer for my team.  The spot’s been open for a while and interviews from ni.com and monster have come up empty.  I had to fill the last two open slots on my team by using a headhunter – expensive.  So I went to Craigslist and posted this under “jobs - internet engineering”:

Web Systems Engineer Wanted For Night Out and Cuddling

It linked to a 'straight' job posting on our corporate site.   Too edgy you say?  Well, it got picked up and blogged about and dugg immediately; people were actively forwarding it around to their friends and emailing me about how cool it was and how cool our company must be as a result.  I could explain why this ad was good but this blogger dude explains it well, so click away.

I used craigslist because it was more expeditious than a full ‘go to the mountain” approach, but I started working on that too, figuring out for conferences Iw as attending if there was a good place to post help wanted where attendees could see them.  I got ten applicants, eight of which were worthy of a phone screen, within two weeks off the craigslist ad.  That’s more phone screens than I’ve ever gotten in a cycle – when I posted this same job in the fall, I got 17 Monster hits, none of which were screenable, and one screenable off a dozen submissions on our corporate site – my other 2 screens were personal referrals.  None passed and one withdrew.  That makes my ratio of resumes to phone screens to interviews to offers to accepts 30:3:0:0:0.  Which sucks, in case you’re not keeping score.

Then I conducted phone screens, using the format Spolsky preferred, with one generic problem question that’s not as vague as a brain teaser and not as specific as “code or command X.”  I thought for a while about the question and asked my team for ideas.  Spolsky’s ideas for questions were more programmer-focused and I wanted one that would tap the systems side more.  My “script” is as follows.

1. “Tell me about yourself”
   1. Listen: Passion – ask about exciting trends
   2. Listen: Problem solver
   3. Delve: Technology – get deep about what they do hands on
   4. Delve: Drive – ask about stuff they personally spearheaded
2. Technical Question:  How would you implement an infrastructure to support an iTunes-like media sales and download site?
   1. Sub-question - "When someone calls and says 'the site is down' where do you go from there?"
   2. Sub-question - What, given your experience, will probably be the top 3 problems that come up post go live?
3. Their Questions
   1. Listen: Preparation
   2. Sell them!  Company, Austin, the team.

After each phone screen I felt that I had a really good idea of their expertise.  The technical question helped – and it’s not a “sit back and let them answer” kind of thing, you conduct it as if the two of you were planning this system together (or more on point, that they’re an employee of yours proposing this design for a system and you’re making sure they know what’s up).

To make a long story short, we got some really good candidates and hires from that round of recruiting.  We got one superstar who was moving back to the States from Germany, but sadly lost him to a competitive offer from a local startup.  Alas.  But on the whole, I think the advice in Joel's book made our recruiting more successful.  It was also helpful justification for the approach - when someone was like "Oh that job post is too freaky" or "You should ask them 100 lame multiple choice questions about what kind of a tree they would be instead of the iTunes thing" I could hand them the book and say "this isn't just me being freaky, it's a best practices tech hiring approach.  Simmer down."