From the “sad but true” files comes an extremely insightful point apparently discussed over beer by the UK devops crew recently – that we are talking about dev and ops collaboration but the current state of collaboration among ops teams is pretty crappy.<\/p>\n
This resonates deeply with me.\u00a0 I’ve seen that problem in spades.\u00a0 I think in general that a lot of the discussion about the agile ops space is too simplistic in that it seems tuned to organizations of “five guys, three of whom are coders and two of whom are operations” and there’s no differentiation.\u00a0 In real life, there’s often larger orgs and a lot of differentiation that causes various collaboration challenges.\u00a0 Some people refer to this as Web vs Enterprise, but I don’t think that’s strictly true; once your Web shop grows from 5 guys to 200 it runs afoul of this too – it’s a simple scalability and organizational engineering problem.<\/p>\n
As an aside, I don’t even like the “Ops” term – a sysadmin team can split into subgroups that do systems engineering, release management, and operational support…\u00a0 Just saying “Ops” seems to me to create implications of not being a partner in the initial design and development of the overall system\/app\/service\/site\/whatever you want to call it.<\/p>\n
Ops Verticals<\/strong><\/p>\n Here, we have a large Infrastructure department.\u00a0 Originally, it was completely siloed by technology verticals, and there’s a lot of subgroups.\u00a0 Network, UNIX, Windows, DBA, Lotus Notes, Telecom, Storage, Data Center…\u00a0 Some ten plus years ago when the company launched their Web site in earnest, they quickly realized that wasn’t going to work out.\u00a0 You had the buck-passing behavior described in the blog posts above that made issues impossible to solve in a timely fashion, plus it made collaboration with devs\/business nearly impossible.\u00a0 Not only did you need like 8 admins to come involve themselves in your project, but they did not speak similar enough languages – you’d have some crusty UNIX admin yelling “WHAT ABOUT THE INODES” until the business analyst started to cry.<\/p>\n Dev Silos<\/strong><\/p>\n But are our developers here better off?\u00a0 They are siloed by business unit.\u00a0 Just among the Web developers there’s the eCommerce developers, eCRM, Product Advisors, Community, Support, Content Management…\u00a0 On the one hand, they are able to be very agile in creating solutions inside their specific niche.\u00a0 On the other hand, they are all working within the same system environment, and they don’t always stay on the same page in terms of what technologies they are using. “Well, I’m sure THAT team bought a lovely million dollar CMS, but we’re going to buy our own different million dollar CMS.\u00a0\u00a0 No, you don’t get more admin resource.”\u00a0 Over time, they tried to produce architecture groups and other cross-team initiatives to try to rein in the craziness, with mixed but overall positive results.<\/p>\n Plugging the Dike <\/strong><\/p>\n What we did was create a Web Administration group (Web Ops, whatever you want to call it) that was holistically responsible for Web site uptime, performance, and security.\u00a0 Running that team was my previous gig, did it for five years.\u00a0 That group was more horizontally focused and would serve as an interface to the various technology verticals; it worked closely with developers in system design during development, coordinated the release process, and involved devs in troubleshooting during the production phase.<\/p>\n B<\/strong>izOps?<\/strong><\/p>\n In fact, we didn’t just partner with the developers – we partnered with the business owners of our Web site too, instead of tolerating the old model of “Business collaborates with the developers, who then come and tell ops what to do.”\u00a0 This was a remarkably easy sell really.\u00a0 The company lost money every minute the Web site was down, and it was clear that the dev silos weren’t going to be able to fix that any more than the ops silos were.\u00a0 So we quickly got a seat at the same table.<\/p>\n Results<\/strong><\/p>\n This was a huge success.\u00a0 To this day, our director of Web Marketing is one of the biggest advocates of the Web operations team.\u00a0 Since then, other application administration (our word for this cross-disciplinary ops) teams have formed along the same model.\u00a0 The DevOps collaboration has been good overall – with certain stresses coming from the Web Ops team’s role as gatekeeper and process enforcement.\u00a0 Ironically, the biggest issues and worst relationships were within Infrastructure between the ops teams!<\/p>\n OpsOps – The Fly In The Ointment<\/strong><\/p>\n The ops team silos haven’t gone down quietly.\u00a0 To this day the head DBA still says “I don’t see a good reason for you guys [WebOps] to exist.”\u00a0 I think there’s a common “a thing is just the sum of its parts” mindset among admins for whatever reason.\u00a0 There are also turf wars arising from the technology silo division and the blurring of technology lines by modern tech.\u00a0 I tried again and again to pitch “collaborative system administration.”\u00a0 But the default sysadmin behavior is to say “these systems are mine and I have root on them.\u00a0 Those are your systems and you have root on them.\u00a0 Stay on your side of the line and I’ll stay on mine.”<\/p>\n Fun specific Catch-22 situations we found ourselves in:<\/p>\n Some of the problem is just attitude, pure and simple.\u00a0 We had problems even with collaboration inside the various ops teams!\u00a0 We’d work with one DBA to design a system and then later need to get support from another DBA, who would gripe that “no one told\/consulted them!”\u00a0 Part of the value of the agile principles that “DevOps” tries to distill is just a generic “get it into your damn head you need to be communicating and working together and that needs to be your default mode of operation.” I think it’s great to harp on that message because it’s little understood among ops.\u00a0 For every dev group that deliberately ostracizes their ops team, there’s two ops teams who don’t think they need to talk to the devs – in the end, it’s mostly our fault.<\/p>\n Part of the problem is organizational.\u00a0 I also believe (and ITIL, I think, agrees with me) that the technology-silo model has outlived its usefulness.\u00a0 I’d like to see admin teams organized by service area with integral DBAs, OS admins, etc.\u00a0 But people are scared of this for a couple reasons.\u00a0 One is that those admins might do things differently from area to area (the same problem we have with our devs) – this could be mitigated by “same tech” cross-org standards\/discussions.\u00a0 The other is that this model is not the cheapest.\u00a0 You can squeeze every last penny out if you only have 4 Windows admins and they’re shared by 8 functional areas.\u00a0 Of course, you are cutting off your nose to spite your face because you lose lots more in abandoned agility, but frankly corporate finance rules (minimize G&A spending) are a powerful driver here.<\/p>\n If nothing else, there’s not “one right organization” – I’d be tempted to reorg everyone from verticals into horizontals, let that run for 5 years, and then reorg back the other way, just to keep the stratification from setting in.<\/p>\n Specialist vs Generalist<\/strong><\/p>\n One other issue.\u00a0 The Web Ops team we created required us to hire generalists – but generalists that knew their stuff in a lot of different areas.\u00a0 It became very hard to hire for that position and training took months before someone was at all effective.\u00a0 Being a generalist doesn’t scale well.\u00a0 Specialization is inevitable and, indeed, desirable (as I think pretty much anything in the history of anything demonstrates).\u00a0 You can mitigate that with some cross-training and having people be generalists in some areas, but in the end, once you get past that “three devs, two ops, that’s the company” model, specialization is needed.<\/p>\n That’s why I think one of the common definitions of DevOps – all ops folks learning to be developers or vice versa – is fundamentally flawed.\u00a0 It’s not sustainable.\u00a0 You either need to hire all expensive superstars that can be good at both, or you hire people that suck at both.<\/p>\n What you do is have people with varying mixes.\u00a0 In my current team we have a continuum of pure ops people, ops folks doing light dev, devs doing light ops, and pure devs.\u00a0 It’s good to have some folks who are generalizing and some who are specializing.\u00a0 It’s not specializing that is bad, it’s specialists who don’t collaborate that are bad.<\/p>\n Conclusion<\/strong><\/p>\n So I’ve shared a lot of experiences and opinions above but I’m not sure I have a brilliant solution to the problem.\u00a0 I do think we need to recognize that Ops\/Ops collaboration is an issue that arises with scale and one potentially even harder to overcome than Dev\/Ops collaboration.\u00a0 I do think stressing collaboration as a value and trying to break down organizational silos may help.\u00a0 I’d be happy to hear other folks’ experiences and thoughts!<\/p>\n","protected":false},"excerpt":{"rendered":" From the “sad but true” files comes an extremely insightful point apparently discussed over beer by the UK devops crew recently – that we are talking about dev and ops collaboration but the current state of collaboration among ops teams is pretty crappy. Internal Borders by Graham Bleach DevOps is a good cause, but what […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[381],"tags":[172,632,255,631,402],"class_list":["post-427","post","type-post","status-publish","format-standard","hentry","category-operations","tag-agile","tag-devops","tag-infrastructure","tag-operations","tag-opsops"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pfI0c-6T","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/427","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=427"}],"version-history":[{"count":1,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/427\/revisions"}],"predecessor-version":[{"id":428,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/427\/revisions\/428"}],"wp:attachment":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=427"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=427"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=427"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n