Web Admin Blog Real Web Admins. Real World Experience.

5Jun/142

When an “Enterprise” Product Isn’t Enterprise Ready

I absolutely love my job and one of the coolest things about what I do is getting to do proof-of-concepts with bleeding edge technology.  I feel very privileged that many companies out there respect me enough to provide me with these opportunities and I feel that engaging on this level enables me to be a better security practitioner because I routinely have my finger on the pulse of the latest and greatest tools out there.  The problem that I run into, however, is when vendors present me "enterprise ready" tools that are clearly not enterprise ready.  Maybe it's a cool concept, a working prototype, or even a functional system.  The problem is that "enterprise ready" assumes so much more than just a product that does some stuff as advertised.  To me, at least, it assumes a product that can be easily transitioned to the company's IT team for long-term support of the platform.   Here are some signs to look out for that will tell you if the tool is truly ready for the big show:

  1. Installation Process: This one could honestly go either way.  Personally, I prefer to have a product that I can install and configure myself.  I cringe every time I hear a vendor mention to me that professional services are involved in an installation.  I get it, sometimes a tool is so highly customized to your environment that you need help, but the majority of the products I use on a daily basis aren't that way.  If installing a product requires a day of professional services time, then this should probably be your first signal to at least start looking out for the following additional signs.
  2. Initialization Script: I honestly feel a bit silly even having to mention this as I would assume this to be a standard part of any enterprise product, but it's not.  If I have to poke around in the installation directory looking for the right script to run to start or stop your product, then it's not enterprise ready.  Even worse, if it's a more complex product that requires starting multiple pieces and you don't have a single init script to handle the startup and shutdown in the proper order, then your product is not enterprise ready.  If you're trying to sell me something to make my life as a security professional easier, then I should spend my time using your tool instead of figuring out how to start and stop it.
  3. Release Notifications: If I buy a product from you and I'm paying you for support, then, I'm typically doing so with the intention that I will be able to move to the next version once it is released.  Maybe it's because there are bugs that need to be fix or because there is new functionality, but whatever the reason, I want to know when that version becomes available.  I'll talk a bit more about the upgrade process itself in the next bullet, but if the company does not have a way to notify you when a new release is available, be wary.
  4. Defined Upgrade Process: Have you ever used a tool that you thought was completely awesome until the first time that an upgrade rolled around?  They tell you copy these files over and it breaks.  Now, run this script and it fails.  You engage support and spend hours on the phone with them and then a week later they offer a webex where a support person will take care of the upgrade for you.  I had to ditch a really interesting tool a while back for this very reason and I'm currently dealing with another one where every upgrade requires a support person to come onsite.  It's a completely ineffective use of both my time and theirs.  When I designed SimpleRisk, one of the first things I considered was how to make it as simple as possible for a remote person to upgrade the tool without assistance.  I've at least got it down to copying some files and running a script which anyone can do.  Even better are the companies where it's click a button to upgrade.  Better still are the companies that just automatically do the upgrade for you.  In any case, be wary of any upgrade processes that are not well-defined.
  5. Backup Plan: This may not apply to all products or all scenarios, but it's a good idea when evaluating a product to ask yourself how you will back up the data and recover it if a disaster ever strikes.  If the answer is "We'd just wipe and reinstall", then cool, but if the answer is "F*ck, I don't know", it may be worth having that discussion with the vendor.
  6. Monitoring: Nothing bothers me more than when I'm all excited to use my shiny new toy and when I go to log in it's down.  In reality, I should know it's down when it happens because there's a high likelihood that the tool isn't doing what it's supposed to if it's not running.  Ask your vendor what you should be monitoring in order to ensure that the tool is functioning properly.  If they don't have a good answer for you, be wary.
  7. Product Roadmap: When you purchase a product, you purchase it not only for what it's capable of doing for you today, but also for the opportunities that it will provide you with tomorrow.  Ask the vendor about their product roadmap to see if it's in-line with your vision of how you intend to use the product.  Are there features that you can use down the line?  More importantly, do they have plans to continue to invest in the platform that they are selling you or is it just major bug fixes at this point while they move on to something else.  If the vendor can't give you a straight answer to this question, then you may have problems.

Don't get me wrong.  There are plenty of tools out there that fail one or more of these signs and that doesn't mean that you should completely avoid them, but you shouldn't expect to pay a premium for them either.  Hopefully the vendor is being honest with themselves and labeling it as "Beta" while they work to iron these things out.  If not, you should be honest with them about your willingness to accept a product that is not "enterprise ready".  Perhaps you're willing to accept a little bit of pain for a smaller price tag.  Maybe you want to be able to brag to your peers that you were the first to have that product hotness.  Whatever the reason, just make sure that you are aware of what you're getting into up front.

3Sep/130

Six Reasons Why Your Company Needs a Chief Information Security Officer (CISO)

I am going to start out here by saying that I do not now, nor have I ever, held the title of Chief Information Security Officer (CISO).  That having been said, I do effectively fill this role as the Information Security Program Owner for a large, $1B+ per year, public company.  Some of what follows will be diatribe on my current role and what I would change if given the opportunity.  Some of it will be based on general observations of how I've seen other companies handle internal security.  What follows are six reasons why your company needs a Chief Information Security Officer (CISO).

Let's start out with how I got my current title.  Early in my career I was a *nix Administrator working for a number of different companies.  I did everything from working as support at a website hosting company to building systems as a military contractor.  Even though my official title never had anything to do with security, I have always had a passion for it, so I always found a way to make it part of my job.  Fast forward to about seven years ago where I got a job as a Web Systems Engineer at my current employer.  I quickly realized that there was nobody handling security for our systems so I decided to shoulder that responsibility.  I began by running Qualys scans, analyzing the results, and fixing the vulnerabilities.  Since this was not my primary role, it all of this work was performed in about 5% of my overall time spent, but I was able to keep good metrics and show fantastic results over time.  After several years of working like this, I finally made the decision to dedicate myself to security full-time.  I got a job offer from another company to be a Security Engineer, but decided to see if my current company was interested in allowing me a similar move.  After some discussions and a few presentations on what the job would entail, I was officially allowed to spend 100% of my time on security.  The only catch was that I was now responsible for our IT SOX testing as well.  Now came the question of what to call my new role.  The title "C" anything is reserved for our executives so that took CISO off the table immediately.  Sad.  I was also not allowed to use the term "Manager" in my title since that indicated having people underneath me.  Even sadder.  I finally settled on "Information Security Program Owner" as it indicated an ownership role in security (as close to Manager as I could get) while staying away from those other non-sanctioned titles.

Alright, so what does any of that have to do with needing a CISO you ask?  To start with, I was the only security professional in the entire company of roughly 5,000 employees at the time.  While officially my purview was in the area of IT security for the enterprise, the lack of any other experts quickly made me a hot commodity.  I was asked to participate on various architecture teams, several teams having to do with regulatory compliance, and even to consult with our R&D teams on product security from time to time.  I'd like to believe that it was because I am so awesome that people couldn't get enough of me, but the God's honest truth is that the entire company had a need and desire for security and there wasn't anyone else to assist.  Which leads me to the first reason why your company needs a Chief Information Security Officer:

Reason #1: By definition, the CISO is where the buck stops as far as security is concerned for your organization.  It is the CISO's job to make sure that security is a concerted effort and that your efforts are not inefficiently duplicated in multiple business units.  Without a CISO, you may may have operational security, but you likely lack direction or a long-term plan for an actual security program.

Now, while my title says "he owns the security program", the fact is that I am not officially a manager or executive.  Thus, on an official level, I pull about as much weight as any other individual contributor in the organization.  It's a precarious position to be in.  On one hand I'm charged with ensuring the security of everyone and everything in the company.  Sometimes this can require being the bad guy and telling people their stuff is broken.  On the other hand, I don't hold enough power to actually force any action that others don't actually want to take.  Perhaps I'll write a future post about how I've managed to still get things done despite this dilemma, but for now this leads me to the second reason why your company needs a Chief Information Security Officer:

Reason #2: Designating one of your senior security resources as the CISO is a form of empowerment.  You are making a statement that they are the person that you trust to make informed security decisions for the organization.  It helps if you can have them report to another C-level executive, like the CFO, but the most important thing here is the title as Chief Information Security Officer says that they are in charge of everything security (everything Information Security if you want to get technical) for your organization.  This helps tremendously in ensuring that security is still a priority when business turns political.

When you hear the title Chief Information Security Officer, what do you think of?  Maybe the IT guy who handle the IPS system?  The guy who goes running around when a system is infected with malware?  Maybe even the guy who wrote the Information Security Policy if you're lucky?  Your CISO should be all these things and so much more.  This leads me to the third reason why your company needs a Chief Information Security Officer:

Reason #3: Your CISO is all things security.  Wikipedia does a great job listing some of the many roles of the Chief Information Security Officer so I'm just going to steal them and list them here:

  • Information Security and Information Assurance
  • Information Regulatory Compliance (PCI, SOX, HIPAA, etc)
  • Information Risk Management
  • Supply Chain Risk Management
  • Cybersecurity
  • Information Technology Controls
  • Information Privacy
  • Computer Emergency Response Team
  • Identity and Access Management
  • Security Architecture
  • IT Investigations, Digital Forensics, and eDiscovery
  • Disaster Recovery and Business Continuity Management
  • Information Security Operations Center
  • PR

Obviously one person cannot handle all of these things which is why most companies have a team of security professionals (ie. Information Security Officers) who report up to the CISO, but this should give you an idea as to the wide scope of what the CISO is responsible for.  Chances are that if you don't have a CISO, then many of these activities aren't happening.  Even worse, the ones that are happening likely aren't aligned with your business objectives.  It's tough to justify spending any money on a program when it performs activities ad-hoc and completely separate from your business.  Which leads me to the fourth reason why your company needs a Chief Information Security Officer:

Reason #4: Your CISO is a business executive that spans into the technical world of security as well.  They should be involved in the business decisions of the company so that they can ensure that the company's security activities are well-aligned with the projects that the business is undertaking.

Hopefully, your senior security professional is an extremely valued member of your team.  If you are holding off on giving them an official Chief Information Security Officer title, then you are doing both them and your company a disservice.  Security companies are organizing events all the time that are targeted at these executives who control the security purse strings.  Sometimes they call them CISO Roundtables, Summits, or otherwise, but the gist of it is that they are a form of education for the CISO and provides them with the opportunity to network with other security professionals in the area, all on somebody else's dime.  The catch is that you're only invited if you're a CISO.  This leads me to the fifth reason why your company needs a Chief Information Security Officer:

Reason #5: The title of CISO is synonymous with "the person in charge of security" for your company and worlds of opportunity open up for them when you bestow upon them that title.  It means free lunches, free trainings, and a host of other perks that unfortunately aren't available with a title like "Security Manager" or "Senior Security Engineer".  Think of it as a job perk that doesn't cost your company a thing.

Before I wrap this up, I have one final reason why your company needs a Chief Information Security Officer, but it's certainly not for everyone.  Occasionally, you'll find a person both technically talented as well as someone who has an affinity and desire to do public speaking.  If this is your senior security person, then it's time to lock them down as they have the ability to do more positive marketing for your company than your entire marketing department.  This leads me to the sixth and final reason why your company needs a Chief Information Security Officer:

Reason #6: If your CISO is willing and able to give engaging talks about security-related topics, then that person, with that title, can make a world of difference for your organization from a marketing perspective.  Conferences are always looking for new and interesting talks and attendees often consider the speakers as industry luminaries.  No marketing whitepaper will ever come close to the exposure potential of having your own industry expert, presenting on a fantastic topic, using a company branded slide deck, in front of hundreds of security professionals.

There you have my six reasons why your company needs a Chief Information Security Officer.  I hope that this was helpful in your search for becoming or designating your company's ultimate CISO.  Feel free to add your own thoughts in the comments below.

29Apr/100

Our First DevOps Implementation

Although we're currently engaged in a more radical agile infrastructure implementation, I thought I'd share our previous evolutionary DevOps implementation here (way before the term was coined, but in retrospect I think it hits a lot of the same notes) and what we learned along the way.

Here at NI we did what I'll call a larval DevOps implementation starting about seven years ago when I came and took over our Web Systems team, essentially an applications administration/operations team for our Web site and other Web-related technologies.  There was zero automation and the model was very much "some developers show up with code and we had to put it in production and somehow deal with the many crashes per day resulting from it."  We would get 100-200 on-call pages a week from things going wrong in production.  We had especially entertaining weeks where Belgian hackers would replace pages on our site with French translations of the Hacker's Manifesto.  You know, standard Wild West stuff.  You've been there.

Step One: Partner With The Business

First thing I did (remember this is 2002), I partnered with the business leaders to get a "seat at the table" along with the development managers.  It turned out that our director of Web marketing was very open to the message of performance, availability, and security and gave us a lot of support.

This is an area where I think we're still ahead of even a lot of the DevOps message.  Agile development carries a huge tenet about developers partnering side-by-side with "the business" (end users, domain experts, and whatnot).  DevOps is now talking about Ops partnering with developers, but in reality that's a stab at the overall more successful model of "biz, dev, and ops all working together at once."

14Jan/102

Book Review: Smart & Gets Things Done, by Joel Spolsky

Joel Spolsky is a bit of an internet cause célèbre, the founder of Fog Creek Software and writer of joelonsoftware.com, an influential programming Web site.

The book is about technical recruiting and retention, and even though it’s a small format under 200 page book, it covers a lot of different topics.  His focus is on hiring programmers but I think a lot of the same principles apply to hiring for systems admin/Web systems positions.  Hiring has been one of the hardest parts of being a Web systems manager, so I got a lot out of the book and tried putting it into practice.  Results detailed below!

The Book

The first chapter talks about the relative effectiveness of programmers.  We often hire programmers and pay the good ones 10% more than the bad ones.  But he has actual data, drawn from a Yale professor who repeatedly teaches the same CS class and assigns the same projects, which shows something that those of us who have been in the field for a long time know – which is that the gap in achievement between the best programmers and the worst ones is a factor of ten.  That’s right.  In a highly controlled environment, the best programmers completed projects 3-4 times faster than the average and 10x faster than the slowest ones.  (And this same relationship holds when adjusting for quality of results.)  I’ve been in IT for 15 years and I can guarantee this is true.  You can give the same programming task to a bunch of different programmers and get results from “Here, I did it last night” to “Oh, that’ll take three months.”  He goes on to note other ways in which you can get 10 mediocre programmers that cannot achieve the same “high notes” as one good programmer.  This goes to reinforce how important the programmer, as human capital, is to an organization.

Next, he delves into how you find good developers.  Unfortunately, the easy answers don’t work.  Posting on monster.com or craigslist gets lots of hits but few keeps.  Employee referrals don’t always get the best people either.  How do you?  He has three suggestions.

  1. Go to the mountain
  2. Internships
  3. Build your own community

“Go to the mountain” means to figure out where the smart people are that you want to hire, and go hang out there.  Conferences.  Organizations.  Web sites.  General job sites are zoos, you need venues that are more specifically spot on.  Want a security guy?  Post on OWASP or ISSA forums, not monster.

We do pretty well with internships, even enhancing that with company sponsored student sourcing/class projects and a large campus recruiting program.  He has some good sub-points however – like make your offers early.  If you liked them as an intern, offer them a full-time job at that point for when they graduate, don't wait.  Waiting puts you into more of a competitive situation.  And interns should be paid, given great work to do, and courted for the perm job during the internship.

Building a community – he acknowledges that’s hard.  Our company has external communities but not really for IT.  For a lot of positions we should be on our our forums like fricking scavengers trying to hire people that post there.