I have been on the older Roadrunner Turbo-charged plan since basically when it first came out and have been generally happy with the service up until recently when I've started having to reboot the modem daily. I'm also kind of an internet speed addict so the idea of moving up to 20 MB/s downloads and 5 MB/s uploads was pretty sweet to me. That's just to start with as eventually the service will have 30 MB/s downloads. I called up Time Warner and asked what it would take to move onto the Extreme Broadband Beta and they told me that it was only an extra $5/mo over my Turbo-charged plan. Even better was that they were offering free installation as part of the Beta. They were able to get the install scheduled just over a week out. Not too bad.

The service technicians came out on the designated day and time and got everything hooked up for me. They even replaced a bunch of the wiring on the box on the side of the house where the service connects to. They did some line tests and within minutes I was up and running on the new service. While not the 5 MB/s upload that was advertised to me, the download speed is quite impressive. Check it out:

The other cool thing is that while not necessarily intended, it is very easy to get into the new ubee modem's configuration interface. By default, the device comes up as 192.168.0.1 on your network and has a username and password of user/user. Get in there and it's got all of the configuration options of a wireless internet gateway. The first thing that you should do is change the username and password. After that, enable the wireless network, configure port forwarding, etc.

Not only does the new modem have built-in wireless N, but it also has four additional network ports so you can use it with multiple computers on your network. I remember the days when Time Warner used to charge you if you had more than one computer, but not anymore.

Granted, I've only had the new service for a few hours now, but I'm already pretty impressed. If you're an internet speed demon like me, and you live in the Austin area, I'd recommend that you give Time Warner a call and ask about switching over to the new Roadrunner Extreme Broadband Beta. Enjoy!

This leads me to the conclusion that security should be considered a fundamental part of code development and not an afterthought. We should be teaching security to students at a University level so that when they graduate, corporations don't spend valuable time re-training them on proper development techniques. I've heard rumors of large companies like Oracle actually being able to impact college curriculum by telling universities they simply won't hire developers without security training. Unfortunately, most companies aren't in a position to make demands like that, but it certainly wouldn't hurt to develop relationships with faculty at your local university and tell them what you'd like to see out of their students. I did some poking around on the internet and it seems like some professors are already starting to get the memo. For example, I found a great paper written by three professors at the USAF Academy Dept. of Computer Science called Incorporating Security Issues Throughout The Computer Science Curriculum where they say:

While the general public is becoming more aware of security issues, what are our universities doing to produce graduates ready to address our security needs?  Computer science as a discipline has matured to the point that students are regularly in tructed in software engineering principles--they learn the importance of life cycle issues in the development and maintenance of software.  Where are they receiving similar instruction on security concerns in the software life cycle?  The authors propose that security should be taught throughout every computer science curriculum--that security should always be a concern and should be considered in the development of all software just as structured programming and documentation are.

Gentlemen, I couldn't agree more.  Security needs to be a foundational piece of every Computer Science program in the country.  Not one class.  Not one professor.  Secure programming techniques need to be a consideration in every CS class in every university.  Universities teach students how to write functions, create object-oriented code, and do proper documentation, but when graduates don't know the basic tenets of input validation, then we have a real problem.  If you agree with me, then I challenge you to write to the Dean of your local CS program and ask them what they are doing to ensure graduates are familiar with secure coding practices.  I'd be very interested in hearing back from you as to what their response was.

I was walking by one of the Iron Mountain Secure Shredding bins at work one day several months ago and noticed that the lock wasn't actually locked. Being the security conscious individual that I am, I tried to latch the lock again, but the lock was so rusted that it wouldn't close as hard as I tried. I can't just leave it there like that so I call the number on the bin's label and there is an automated message that tells me that they're not taking local calls anymore and gave me a different number to try. I call that number and they ask me for my company ID number which I had no idea what it was. She informed me that without that ID number I couldn't submit a support request. I informed the lady that this bin contained sensitive personal and financial information and that the issue couldn't wait for some random company ID to be found. Fortunately, she gave in and created the support ticket for me saying that I should hear back from someone within four hours.

One week later, on Friday, Iron Mountain finally calls me back and says that they will come to replace the lock the following Monday before 5 PM. When the lock hadn't been replaced yet on Monday evening, I called Iron Mountain back up. Looking at their records, they showed that a new lock had been delivered, but they had no idea where and the signature was illegible. I work on a three-building campus with 14 floors between them and almost 3,000 people. If they can't tell me where the lock is, then there's no way for me to track it down. They said that they would investigate and call me back.

After not hearing back from them again for a couple of days, I called them back. The woman I spoke with had no real update on the investigation. She said that she would send another message "downstairs" and escalate to her supervisor. At this point it had been almost three weeks with sensitive documents sitting in a bin with a malfunctioning lock. The next day they called me back and said they were never able to track down who the new lock was left with so they would bring us a new one at no charge. Finally, after a total of 24 days with a unlocked Secure Shredding bin, Iron Mountain was able to replace the lock. Iron Mountain......FAIL.

I wanted to make sure I fully understood what was going on under the hood here so I started asking questions about the static testing and how it works. They've got a nice looking portal where you name your application, give it a version, assign it to a group of developers, and point it to your compiled code (WAR, EAR, JAR, etc). Once you upload your binaries, their system basically runs a disassembler on it to get it into assembly code. It's then at this level that they start looking for vulnerabilities. They said that this process takes about 3 days initially and then maybe 2 days after the first time because they are able to re-use some data about your application. Once complete, they say they are able to provide you a report detailing your vulnerabilities and how to fix them.

The thing that immediately struck me as worth noting here was the 2-3 day turnaround. This means that our developers would need to wait a fairly substantial amount of time before getting any feedback on the vulnerability status of their code. In a world full of Agile development, 2-3 days is a lifetime. Compare that to static source code testing where you get actionable results at compile time. The edge here definitely goes to source code testing as I believe most people would prefer the near-instant gratification.

The next thing worth noting was that they are taking binary files and disassembling them in order to find vulnerabilities. This lends itself to one major issue which is how can you determine with any accuracy the line number of a particular vulnerability written in let's say Java from assembly code generated by disassembling the binaries. By default, it's simply not possible. This vendor claimed that they can by adding in some debug strings at compile time, but even then I'd contend that you're not going to get much. I'm guessing they have some heuristics that are able to tell what function generated a set of assembly code, but I'm extremely skeptical that they can do anything with variable names, custom code functions, etc. I've seen some source code scanners, on the other hand, that not only tell you what line of code is affected, but are able to give you an entire list of parameters that have been consequently affected by that vulnerability. The edge here definitely goes to source code testing.

The main benefit that I can see with binary testing vs source code testing is that we can test code that we didn't write. Things like APIs, third-party applications, open source, etc are all things that we now have visibility into. The only problem here is that while we now can see the vulnerabilities in this software, they are unfortunately all things that we can't directly influence change in, unless we want to send our developers off to work on somebody else's software. I'd argue that scanning for vulnerabilities in that type of code is their responsibility, not ours. Granted, it'd be nice to have validation that there aren't vulnerabilities there that we're exposing ourselves to by uptaking it, but in all honesty are we really going to take the time to scan somebody else's work? Probably not. The edge here goes to binary testing with the caveat being that it's in something that I frankly don't care as much about.

This isn't the complete list of pros and cons by any means. It's just me voicing in writing some concerns that I had about the technology while talking to this particular vendor. In my opinion, the benefits of doing source code testing far outweigh any benefits that we could get from testing compiled binary files. What do you think about the benefits of one versus the other? I'd certainly love for someone to try to change my mind here and show me where the real value lies in binary testing.

As a case in point, we're in the middle of our annual audit by one of those "Big Four" audit firms which I won't name here to protect the innocent. I sent an email checking in with our auditors to make sure that they had everything they needed before we went into our four-day holiday weekend. They said that they had received everything they needed except for documentation on "privileged users from the current OS and Database environments" as well as "evidence of current password settings from the application servers, OS, and Database". We go through a round of translation from Auditorese to Techie and figure out that they want exports of some specific user, profile, role, and privilege tables from the database and copies of /etc/passwd, /etc/shadow, and /etc group from the servers.

So we obtain the requested documentation and I shoot them back an email message to find out their proposed method for transferring the files. Secure FTP? No. PGP encryption? Nope. Their response back was astonishing:

How large do you think they'll be? Email should be fine.

Seriously? These are the guys that we're paying to verify that we're properly protecting our systems and they're suggesting that sending our usernames and password hashes via cleartext email is an appropriate method of file transfer. I respond back:

I'm not really concerned about the size of the files, but rather, the data that they contain. Sending files containing the users, groups, and password hashes for our financial systems via cleartext is probably not a good plan considering the point of this process is protecting that data.

And they respond with:

Whatever you'd like Josh. As long as you have the files as of today, we're good.

So now I'm convinced that auditors (or at least these auditors) view security as nothing more than a checklist. The people telling me what I need to do in order to protect my systems really have no clue about the fundamentals of security. If it's not on their checklist, then it must not be of importance. In this particular situation it may be easier or more convenient to send the documents via email, but any security professional worth their salt would tell you that's not secure nor appropriate for that data. Either our auditors hold themselves to a very different standard than the rest of us security professionals or they just don't understand security unless it's on a checklist.

Here at NI we did what I'll call a larval DevOps implementation starting about seven years ago when I came and took over our Web Systems team, essentially an applications administration/operations team for our Web site and other Web-related technologies.  There was zero automation and the model was very much "some developers show up with code and we had to put it in production and somehow deal with the many crashes per day resulting from it."  We would get 100-200 on-call pages a week from things going wrong in production.  We had especially entertaining weeks where Belgian hackers would replace pages on our site with French translations of the Hacker's Manifesto.  You know, standard Wild West stuff.  You've been there.

Step One: Partner With The Business

First thing I did (remember this is 2002), I partnered with the business leaders to get a "seat at the table" along with the development managers.  It turned out that our director of Web marketing was very open to the message of performance, availability, and security and gave us a lot of support.

This is an area where I think we're still ahead of even a lot of the DevOps message.  Agile development carries a huge tenet about developers partnering side-by-side with "the business" (end users, domain experts, and whatnot).  DevOps is now talking about Ops partnering with developers, but in reality that's a stab at the overall more successful model of "biz, dev, and ops all working together at once."

Of course we had ongoing relationships with the development managers, but it was obvious within a very short timeframe that wasn't going to be the end solution.  Sure, there was a little ossified opinion syndrome about the role of Ops (they are the blue collar "dirty people," as we all know) in relation to the developers - "You just move files, how hard is it?" - but mostly it was that our dev groups take a huge amount of very direct guidance and project management from our biz teams, and they would always be telling us "I don't have time to load test and my business analyst doesn't think it's important and says we have to release this week and I just do what they tell me."  The devs weren't empowered in and of themselves to do a lot of the stuff we needed them to do.  That made it critical to get direct business engagement.  Some environments (like my new group) have extremely empowered dev teams that are basically writing specs themselves; in that case probably starting with dev partnership would work well.

I'll also note that being aligned with the business helped us know what we should work on.  One year they'd be more interested in performance, another they'd be more interested in availability, etc.  That gave us the opportunity to be heroes - for example, when we switched over from our old load balancing software to our Citrix Netscalers and our global Web site performance improved by 50%.  We demonstrated "hey, some things Ops can give you, no programmers needed."  So then to a business guy you're just as valuable as a programmer.   They just want people to give them what they want, and if that's you then you're sitting pretty.

Step Two: Automate Some Stuff

Here's where I have to give huge credit to my partner in crime Peco.  I was making sure we were fixing root causes of problems and focusing on metrics to get us more people hired/permission to spend group time on improvement projects to ease the hundred pages a week problem.   Peco's a code hacker at heart so he quickly hit the two biggest pain points we needed to automate, Java application deployment and automated response to pages.  Our app deployment was manual up to that point - devs would hand over long incoherent install docs and we'd go perform them manually on multiple redundant servers.  He quickly wrote some scripts and an SSH framework and we made the developers structure their artifacts and fill out a config file that allowed us to take their apps from revision control and deploy them in an automated fashion to redundant dev, test, or production systems.  The devs loved it - we had a couple yap for the first week about "but but I like .wars not .ears" but the immense benefits became immediately apparent to them, largely thanks to our already pretty mature change control/release process.  Only ops could do deploys to test and we had big monthy Friday night releases for production code that devs had to attend, and dev time spent messing with dev deploys, waiting for test deploys, and staying up all night for production deploys plummeted.  A month later it was one of those "it's always been this way" steps.

After that we always wished we had time to go to a fuller automated CM solution, and as things like puppet and chef appeared we eyed them with interest, but we stemmed a huge part of the tide with that one Perl program.

Peco also wrote a framework to integrate with our monitoring solution and go do things like 'restart the server' without operator intervention.  Sure, there's down sides to that, but when you have people quitting your team because their oncall week involves restarting app servers all night every night, it serves a clear purpose.

These targeted automation steps moved us from chronic firefighting mode and pushed us over the hump to where we could actually try to make some more powerful changes.

Step Three: Value Added Services

Now that we had a handle on operations, we started to look for areas that no one was really addressing that needed help.  Performance/availability and security were clear weak spots and we had people with expertise and energy in both.  So we started what we called 'practice areas' and started in on tools and training in those areas. First we focused on detection and metric collection - showing the real performance of our Web site and individual apps and scanning for security holes.

The partnering with the business and dev managers was bearing fruit, in fact a Web steering project was established that had as its primary business goals for IT performance, availability, efficiency, TCO, and agility.  This created pressure on individual app teams to address the problems we could now detect, and we were ready with tools to help them fix it - for example, a shout out to Opnet's Panorama, an excellent application performance management (APM) tool that collects OS, software, and deep-dive Java metrics from across a distributed system and correlates those metrics.  We showed them "look, your app running in production isn't a mystery - you can get to it and look into it with this tool, and when there's a slowdown or failure you can get to where it is."  Similary, we implemented Splunk and gave developers constant access to their own log files.

From one point of view, it was us "passing the buck."  "Here, look at your own log files."  But the truth is, that the developers are the ones that wrote those cryptic error messages and know what they mean, and which errors are "normal" (to this day, whenever a developer tells me an error in their log file is "normal" I have to restrain the urge to punch them in the face).  By taking the tack of empowering devs to inspect their apps in production, in conjunction with producing pressure for them to improve operational aspects of their software,we were able to get the right people to work on the problems to really fix them.

Step Four: A Process Emerges

By this time the group had grown from the 2-3 people it started with to 7 people, including an operations guy in Hungary for follow the sun support.  In general we hired only high quality people; some were more experienced than others but we had high skills and esprit de corps.

There started to be a lot of pressure from production support duties interrupting project work.   Once we got large enough we split into a project subteam and an ops subteam to try to prevent that (what O'Reilly's Time Management for System Administrators book calls "interrupt shielding").  We did it as a "rotation" where admins would go onto "tech rotation" for 3 months and then back into projects, so that we could keep developing everyone and keep it "one team" and not "two teams" (no conceptual barrier causing finger pointing, communication hassle...)    We had talked about this when we were 4 people strong, and that wasn't enough people to make it tenable - and if we had grown to 10+ people I'd split it into two permanent groups, but for that middle area this model worked for us.

From the beginning, we had been advocating the idea of "you need to come and get an admin assigned to your project at kickoff, who will work with you to get all the systems stuff worked out."  This worked OK but was hit or miss - there was no formal project initiation process to hook into and so sometimes it happened, sometimes it didn't, sometimes it happened late.  But on projects where it happened, we had a proto-DevOps style ops person working with devs on a specific project.  Sadly though since there were so many projects, usually it wasn't embedded enough to work truly hand in hand (one ops person would be working on 5 projects at a time...).

Over time we did this enough that patterns began to emerge - what things need to happen in different kinds of engagements.  We then created a systems development process where each project needs to engage a Web Admin as part of their project, and it has a detailed playbook on what both the systems engineer and the developers need to do to make sure all the performance/availability/deployment/security/etc "nonfunctional requirements" are addressed.

Complications: Things That Didn't Work Well

Keep in mind we didn't have the full DevOps vision in mind when we started, we were evolving in the direction that seemed right to us as we got the opportunity over time.  So this all happened over years, whereas now I think with this blueprint in mind it could be implemented much more quickly even in an existing org.  These were the most significant issues that recurred while we were trying to do all this.

1. The Web Admins are a Web application administration group, but has to interact with other teams in our pretty large infrastructure group (UNIX admins, Windows admins, DBAs, Notes admins, network admins...  We tried to represent all the infrastructure groups to the programmers but, you know ops groups, there were constant outbreaks of squabbling.  A talk about this a bit in "Before DevOps, Don't You Need OpsOps?"  For a variety of reasons our relationships with the business (usually very good) and the devs (good and rocky alternating) were always better than our relationships with the other Infrastructure groups (always grumpy).
2. Our software dev process hasn't been very mature traditionally and has tended to scope projects down to single programmer size.  This has been changing over the last 2 years as they have embraced agile but there's still a tendency to have "many small projects."  As there were about 50 programmers and about 7 Web Admins (of which only 5 were available for projects), that caused a significant problem with trying to really partner with them on projects.  We tried focusing on "only large projects", but still you'd have projects where the ops person just didn't give the project enough time and attention.  And we uptook a lot of major complex platforms (Vignette VCM, Oracle SOA, etc.) that ended up leeching off a pretty constant amount of admin time to master and support (.5 person per platform in the steady state).  So the desire to do full DevOps partnering was always crushed under the wheels of a 1:10 ops:dev ratio.
3. It was also extremely harsh at a management level - our corporate culture, at least in IT, is very "relationship based" and not strongly management-driven.  I was used to "transactional based" organizations, where you go to a group and say "I need a something that is your group's job, so here's the request" and they do it.  Here you kinda have to go ask/sweet-talk/convince other groups to perform tasks.  Which is fine, it's an approach that other "consensus-driven" companies have scaled up with (HP, for example).  But in this role, we had to maintain relationships with a truly huge number of teams - a bunch of infrastructure teams, a bunch of Web programmer teams, and a bunch of Web business teams.  That level of serving/being a customer of 15 or so teams was just barely sustainable, but when we got tapped with running some internal systems like our company's SOA suite and internal collaboration system, and ten other customer and programming teams got dropped into the mix, it broke down (or at least that's where I reached the limit of my modest management skills).
   I think this is a place agile often hits rough patches - you bring in people from various disciplines to get the job done, but if the company's inherent organizational structure requires many separate org structures to do negotiation and resource management to make that work, you hit management paralysis (since the number of relationships is a factorial of the number of teams involved).  To scale you have to put some kind of lubricating process in place to enable that to happen, and we didn't have that.

Conclusion

We weren't operating in an agile process (iterations), but we were pretty successful at several important aspects of agile, especially those people usually mean when they say DevOps - partnering with business and developers, automating routine tasks, etc.  And we definitely build a large, skilled, and respected team that continually brought new tools and practices into the organization.  A lot of lessons learned that tell me that agile operations/DevOps is definitely the way to go, and now that we're getting to implement systems and processes from scratch (more on that later!) we have a lot of understanding of what we need to do/not do.

Killer quote:

"I say this as somebody who about 15 years ago chose system administration over development.  But system administration and system administrators have allowed themselves to lag in maturity behind what the state of the art is. These new technologies are finally causing us to be held to account to modernize the way we do things.  And I think that’s a welcome and healthy challenge."

Some customers have asked us about ways to easily convert virtual machines from VMware vSphere, Citrix Xen Server, and Microsoft Hyper-V to Amazon EC2 instances - and vice versa. If this is something that you're interested in, we would like to hear from you. Please send an email to aws-vm@amazon.com describing your needs and use case.

I'll share my reply here for comment!

This is a killer feature that allows a number of important activities.

1.  Product VMs.  Many suppliers are starting to provide third-party products in the form of VMs instead of software to ease install complexity, or in an attempt to move from a hardware appliance approach to a more-software approach.  This pretty much prevents their use in EC2.  <cue sad music>  As opposed to "Hey, if you can VM-ize your stuff then you're pretty close to being able to offer it as an Amazon AMI or even SaaS offering."  <schwing!>

2.  Leveraging VM Investments.  For any organization that already has a VM infrastructure, it allows for reduction of cost and complexity to be able to manage images in the same way.  It also allows for the much promised but under-delivered "cloud bursting" theory where you can run the same systems locally and use Amazon for excess capacity.  In the current scheme I could make some AMIs "mostly" like my local VMs - but "close" is not good enough to use in production.

3.  Local testing.  I'd love to be able to bring my AMIs "down to me" for rapid redeploy.  I often find myself having to transfer 2.5 gigs of software up to the cloud, install it, find a problem, have our devs fix it and cut another release, transfer it up again (2 hour wait time again, plus paying $$ for the transfer)...

4.  Local troubleshooting. We get an app installed up in the cloud and it's not acting quite right and we need to instrument it somehow to debug.  This process is much easier on a local LAN with the developers' PCs with all their stuff installed.

5.  Local development. A lot of our development exercises the Amazon APIs.  This is one area where Azure has a distinct advantage and can be a threat; in Visual Studio there is a "local Azure fabric" and a dev can write their app and have it running "in Azure" but on their machine, and then when they're ready deploy it up.  This is slightly more than VM consumption, it's VMs plus Eucalyptus or similar porting of the Amazon API to the client side, but it's a killer feature.

Xen or VMWare would be fine - frankly this would be big enough for us I'd change virtualization solutions to the one that worked with EC2.

I just asked one of our developers for his take on value for being able to transition between VMs and EC2 to include in this email, and his response is "Well, it's just a no-brainer, right?"  Right.

I missed the first one but loved OpsCamp so I'm going!