The book is about technical recruiting and retention, and even though it’s a small format under 200 page book, it covers a lot of different topics.  His focus is on hiring programmers but I think a lot of the same principles apply to hiring for systems admin/Web systems positions.  Hiring has been one of the hardest parts of being a Web systems manager, so I got a lot out of the book and tried putting it into practice.  Results detailed below!

The Book

The first chapter talks about the relative effectiveness of programmers.  We often hire programmers and pay the good ones 10% more than the bad ones.  But he has actual data, drawn from a Yale professor who repeatedly teaches the same CS class and assigns the same projects, which shows something that those of us who have been in the field for a long time know – which is that the gap in achievement between the best programmers and the worst ones is a factor of ten.  That’s right.  In a highly controlled environment, the best programmers completed projects 3-4 times faster than the average and 10x faster than the slowest ones.  (And this same relationship holds when adjusting for quality of results.)  I’ve been in IT for 15 years and I can guarantee this is true.  You can give the same programming task to a bunch of different programmers and get results from “Here, I did it last night” to “Oh, that’ll take three months.”  He goes on to note other ways in which you can get 10 mediocre programmers that cannot achieve the same “high notes” as one good programmer.  This goes to reinforce how important the programmer, as human capital, is to an organization.

Next, he delves into how you find good developers.  Unfortunately, the easy answers don’t work.  Posting on monster.com or craigslist gets lots of hits but few keeps.  Employee referrals don’t always get the best people either.  How do you?  He has three suggestions.

1. Go to the mountain
2. Internships
3. Build your own community

“Go to the mountain” means to figure out where the smart people are that you want to hire, and go hang out there.  Conferences.  Organizations.  Web sites.  General job sites are zoos, you need venues that are more specifically spot on.  Want a security guy?  Post on OWASP or ISSA forums, not monster.

We do pretty well with internships, even enhancing that with company sponsored student sourcing/class projects and a large campus recruiting program.  He has some good sub-points however – like make your offers early.  If you liked them as an intern, offer them a full-time job at that point for when they graduate, don't wait.  Waiting puts you into more of a competitive situation.  And interns should be paid, given great work to do, and courted for the perm job during the internship.

Building a community – he acknowledges that’s hard.  Our company has external communities but not really for IT.  For a lot of positions we should be on our our forums like fricking scavengers trying to hire people that post there.

Chapter 3 is interesting; it’s a “Field Guide to Developers.”  It seems obvious, but to get the best developers working for you, you need to be showing that you’re the kind of place they want to work.  What does the wily developer like?

First, there’s the physical workspace.  Does it look cool or awful?  You have to turn around and take a look at yourself as an outsider would.  A big monochrome cube farm, with crappy chairs and computers, and a lobby with wilting plants in it is a warning sign to developers to stay away.   Private offices with Aeron chairs and dual monitors say “no really, we do consider our people more than interchangeable parts.”  You can consider these kinds of things luxuries, but good programmers can go a hundred other places that have them.  We only bat about .500 on this, and that’s mostly due to our beautiful campus.

Side note - we had one guy interview here, who when asked what he found important about a job, replied "a comfy chair and nice monitor."  It's OK for that to be important to you but when it's first thing out in an interview, it makes you seem like you are an entitlement fan and don't really care about your work - we passed on that guy.  Right?  Wrong?  Chime in with a comment.

Next, there’s the interpersonal environment.  Are programmers treated well in the organization, or are they typists given orders from on high?  What are their colleagues like – when they meet people during the interviews, they are always asking themselves “do I want to work with this person?”  Grumpy, bedraggled, unprepared, or distracted says “no.”  He notes at Fog Creek they started with the old Microsoft hiring maxim of “Smart, and gets things done” but soon learned that they (unlike Microsoft) needed to add a third clause, “Not a jerk.”  Does it look like they’ll be given responsibility and autonomy?  Is the environment rife with politics?  Will they be working with cool new technology, on something interesting?   Can they identify with the company or otherwise be attracted by its goals?  Even “side” goals like a wellness program or recycling drive count here.  For anyone who hasn’t noticed, this is the “me” generation.  Will I like working there, will it be happy and beneficial for me day in, day out?

He also notes another truism that too many people don’t believe, which is what programmers don’t care about – money.  (Within reason of course.)  It’s very low on the list and usually only comes up when the other factors are seen as undesirable.

“Sorting Resumes” is the topic of Chapter 4.  He acknowledges how terrible a tool the average resume and cover letter are in terms of conveying anything you really want to know about a candidate.  You can’t hire from them, you can only screen out people you clearly don’t want to hire from them.

Joel’s criteria for sorting resumes are:

• Passion – evidence that they love programming, based on a variety of factors – into computers from an early age, extracurricular computer activities (like working on open source projects), interest in new technologies.
• Pickiness – I call this “fit” - is the applicant looking to work for you?  Did they bother to customize the cover letter?  Remember it’s not all about whether you like them; if you extend an offer you want to have a high probability that they’ll accept.
• English – if your communication skills are so poor that you can’t even get a resume right, when you have an infinite amount of time to craft it and get other people to proofread it, you’re trouble.
• Brains – anything indicating they’re just flat smart.
• Selectivity – if they went through a highly selective process in the past (getting into an Ivy League school, hired by Google) then someone else has pre-vetted them for you.
• Hard-core – certain technologies are harder than others.  If someone’s done C++ programming, they are in general going to have more chops than someone who’s only programmed JavaScript, even if you’re looking for a JavaScript programmer.
• Diversity – not gratuitous diversity, but you want to make sure and have different types of people on your teams.  If your team is all old fogeys, hire someone right out of school so that you can benefit from a different viewpoint.

He also talks about what not to do.  He’s all about making them show some programming chops during an interview, but putting an additional hoop of a technical test into the prequalification process just loses you people who think that’s lame and they can get a good job without being your trained monkey.  He also notes it’s important not to look too much for experience with specific technologies.  The “keyword matching” school of programmer selection is awful.  A good programmer that knows 3 languages can pick up whatever language number 4 you use without any problem, whereas some goon who only knows language 4 will, after about two weeks, be less productive than the other guy forever.  Every once in a while you are specifically looking for an expert, in which case that’s fine – if you want a Java architect for the department then you do want in depth expertise in it.

Chapter 4 covers the phone screen.  He recommends starting with a phone screen, because it actually allows you to do a first cut more “fairly” without seeing the candidate.  And it’s less expensive than an in-person.  His phone interviews have three parts.  The first is the ”tell me about yourself” part.  In this he listens for signs of passion and being a problem solver, and drills down into technology and politics.  By technology, he means establishing exactly what role they played with the technologies on their resume and in their story, to find their real level of technical skill.  And by politics, he really means drive and persistence – how they handle challenges and get things done.

Next, he poses an open-ended technical question to get in tune with their thought processes.  Not looking for pseudocode as an output, but for their ability to make tradeoffs, understand subtleties, etc.  He gives examples like “How would you design a program that lets people play Monopoly with each other over the Internet?”   Something they’d be familiar with but not have implemented.  The key premise here is that smart people can tell other smart people by having an in depth discussion with them over a problem at hand.  Again, having the same problem posed to all candidates allows for quality compare/contrast.

The third and final part of his phone screens are letting the candidate interview him, to aggressively sell them on the position and also to see if they’ve done any basic research into the company, which reveals how serious they are.  Remember, it’s all about closing the deal, so whether you want them is only half of the equation ; are they seriously looking as well and do they want you.

Ernest’s Conclusion: This book is the God’s honest truth and you would do well to read it and follow it.  Sure, he has a couple specific hang-ups that can safely be ignored – how cool iPods are, how important private offices are, and that programmers should fly first class – but every core idea here is sound and proven over and over again in practice.

Let's Use What We've Learned

So let’s go through his process as a test case.  I need a Web Systems engineer for my team.  The spot’s been open for a while and interviews from ni.com and monster have come up empty.  I had to fill the last two open slots on my team by using a headhunter – expensive.  So I went to Craigslist and posted this under “jobs - internet engineering”:

Web Systems Engineer Wanted For Night Out and Cuddling

It linked to a 'straight' job posting on our corporate site.   Too edgy you say?  Well, it got picked up and blogged about and dugg immediately; people were actively forwarding it around to their friends and emailing me about how cool it was and how cool our company must be as a result.  I could explain why this ad was good but this blogger dude explains it well, so click away.

I used craigslist because it was more expeditious than a full ‘go to the mountain” approach, but I started working on that too, figuring out for conferences Iw as attending if there was a good place to post help wanted where attendees could see them.  I got ten applicants, eight of which were worthy of a phone screen, within two weeks off the craigslist ad.  That’s more phone screens than I’ve ever gotten in a cycle – when I posted this same job in the fall, I got 17 Monster hits, none of which were screenable, and one screenable off a dozen submissions on our corporate site – my other 2 screens were personal referrals.  None passed and one withdrew.  That makes my ratio of resumes to phone screens to interviews to offers to accepts 30:3:0:0:0.  Which sucks, in case you’re not keeping score.

Then I conducted phone screens, using the format Spolsky preferred, with one generic problem question that’s not as vague as a brain teaser and not as specific as “code or command X.”  I thought for a while about the question and asked my team for ideas.  Spolsky’s ideas for questions were more programmer-focused and I wanted one that would tap the systems side more.  My “script” is as follows.

1. “Tell me about yourself”
   1. Listen: Passion – ask about exciting trends
   2. Listen: Problem solver
   3. Delve: Technology – get deep about what they do hands on
   4. Delve: Drive – ask about stuff they personally spearheaded
2. Technical Question:  How would you implement an infrastructure to support an iTunes-like media sales and download site?
   1. Sub-question - "When someone calls and says 'the site is down' where do you go from there?"
   2. Sub-question - What, given your experience, will probably be the top 3 problems that come up post go live?
3. Their Questions
   1. Listen: Preparation
   2. Sell them!  Company, Austin, the team.

After each phone screen I felt that I had a really good idea of their expertise.  The technical question helped – and it’s not a “sit back and let them answer” kind of thing, you conduct it as if the two of you were planning this system together (or more on point, that they’re an employee of yours proposing this design for a system and you’re making sure they know what’s up).

To make a long story short, we got some really good candidates and hires from that round of recruiting.  We got one superstar who was moving back to the States from Germany, but sadly lost him to a competitive offer from a local startup.  Alas.  But on the whole, I think the advice in Joel's book made our recruiting more successful.  It was also helpful justification for the approach - when someone was like "Oh that job post is too freaky" or "You should ask them 100 lame multiple choice questions about what kind of a tree they would be instead of the iTunes thing" I could hand them the book and say "this isn't just me being freaky, it's a best practices tech hiring approach.  Simmer down."

What's Changed?

• It's about Risks, not just vulnerabilities
  • New title is: "The Top 10 Most Critical Web Application Security Risks"
• OWASP Top 10 Risk Rating Methodology
  • Based on the OWASP Risk Rating Methodology, used to prioritize Top 10
• 2 Risks Added, 2 Dropped
  • Added: A6 - Security Misconfiguration
    • Was A10 in 2004 Top 10: Insecure Configuration Management
  • Added: A8 - Unvalidated Redirects and Forwards
    • Relatively common and VERY dangerous flaw that is not well know
  • Removed: A3 - Malicious File Execution
    • Primarily a PHP flaw that is dropping in prevalence
  • Removed: A6 - Information Leakage and Improper Error Handling
    • A very prevalent flaw, that does not introduce much risk (normally)

1. A1- Injection: Tricking an application into including unintended commands in the data sent to an interpreter. (http://www.owasp.org/index.php/SQL_Injection_Prevention_Cheat_Sheet)
2. A2 - Cross Site Scripting (XSS): Raw data from attacker is sent to an innocent user's browser.  For large chunks of user supplied HTML, use OWASP's AntiSamy to sanitize this HTML to make it safe.  (http://www.owasp.org/index.php/XSS_(Cross_Site_Scripting)_Prevention_Cheat_Sheet)
3. A3 - Broken Authentication and Session Management: Means credentials have to go with every request.  Should use SSL for everything requiring authentication.
4. A4 - Insecure Direct Object Reference: This is part of enforcing proper "Authorization", along with A7 - Failure to Restrict URL Access.
5. A5 - Cross Site Request Forgery (CSRF): An attack where the victim's browser is tricked into issuing a command to a vulnerable web application.  Vulnerability is caused by browsers automatically including user authentication data with each request.  (Check out OWASP CSRFGuard, OWASP CSRFTester, http://www.owasp.org/index.php/CSRF_Prevention_Cheat_Sheet)
6. A6 - Security Misconfiguration: All through the network and platform.  Don't forget the development environment.  Think of all the places your source code goes.  All credentials should change in production.
7. A7 - Failure to Restrict URL Access: This is part of enforcing proper "authorization", along with A4 - Insecure Direct Object References.
8. A8 - Unvalidated Redirects and Forwards: Web application redirects are very common and frequently include user supplied parameters in the destination URL.  If they aren't validated, attacker can send victim to a site of their choice.
9. A9 - Insecure Cryptographic Storage: Storing sensitive data insecurely.  Failure to identify all sensitive data.  Failure to identify all the places that this sensitive data gets stored.  Failure to properly protect this data in every location.
10. A10 - Insufficient Transport Layer Protection

OWASP Top 10 Risk Rating Methodology

• Attack Vector (How hard for an attacker to use this flaw - 1 (Easy), 2 (Average), 3 (Difficult))
• Weakness Prevalence (How often is it found - 1 (Widespread), 2 (Common), 3 (Uncommon))
• Weakness Detectability (How hard is it for an attacker to find the flaw - 1 (Easy),  2 (Average), 3 (Difficult))
• Technical Impact (1 (Severe), 2 (Moderate), 3 (Minor))

This is generic across the internet, not specific to any organization.

Started a new "Prevention Cheatsheet Series" that the Top 10 references (XSS, SQL Injection, Transport Layer Security, CSRF, Direct Object Reference).

What is actually being released is RC1 of the Top 10 and they are encouraging people to provide comments through the end of the year and then use that feedback to post the final Top 10 in January 2010.

What is log management?

• Ensuring your enterprise log data is accessible, easily retrievable and forensically sound
• Properly dealing with mammoth amounts of event data stores in thousands of vendor generated log files
• Achieving compliance (SOX, HIPAA, PCI, FISMA), Security and IT operation usage of log data that does not break the bank
• Log data now represents over 30% of ALL data generated by enterprises – creating a real need for log management
• Dominant uses for log data include:
  • IT operations – systems/network health and availability
  • Security monitoring – perimeter or insider threat detection
  • Compliance monitoring – for regulations and industry standards

Why should I care?

• Overwhelming flood of logs
• Islands of defense
• Week long manual investigations
• Massive false positives
• Heterogeneous consoles
• Many different formats
• Regulations and their commonly used frameworks impose various requirements when it comes to log management
• Regulatory mandates have further increased log retention requirements
• Increased need to store both security and non-security
• There continues to be an increased emphasis on audit quality data collection
• Regulatory requirements
  • SOX: 7yrs
  • PCI: 1yr
  • GLBA: 6yrs
  • EU DR Directive: 2yrs
  • Basel II: 7yrs
  • HIPAA: 6/7yrs
• Compliance requirements
  • More logging
  • More types of devices
  • Higher volumes of log data
  • Extensive reporting requirements
  • Broader user access
  • Long term retention requirements
  • Audit quality data

What can effective log management do for me?

• Self-managing & scalable
• Automated & cost-effective audits
• IT Operations SLA Efficiency
• Compliance
• Simplified Forensic Investigations

Best Practices – NIST 800-92

• Common log management problems
  • Poor tools and training for staff
  • Laborious and boring
  • Reactive analysis reduces the value of logs
  • Slow response
• Solutions
  • Establish log management policies & procedures
  • Prioritize log management appropriately
  • Create and maintain a secure log management infrastructure
  • Provide proper support for all staff with log management responsibilities
  • Establish standard log management processes for system-level admins
• The directive to only log and analyze data this is of the greatest importance helps provide sanity to the logging process
• Collecting and storing all data regardless of its usefulness increases complexity and deployment costs
• Secure storage and transmission guideline directly points to the importance of secure and robust capture, transmission and storage of logs
• Organizations should carefully review the collection architecture, transmission security and access control capabilities of SEM solutions to ensure support of this section of the standard
• Filtering and aggregation are recommended as a means to only capture logs of security and compliance value based on the corporate retention policy
• Guideline helps organizations support a “reasonableness” position in not collecting useless log data

Developing a Log Management Program

• Understand your log management needs (regulatory and operational requirements)
• Review NIST 800-92
• Understand your environment
  • Lots devices to collect logs from
  • Multiple locations with no IT staff
  • Collection agents are not an option
  • Network time settings
  • Low bandwith links
• Devices
  • Firewalls/VPN
  • IDS/IPS
  • Servers and desktop OS
  • Network equipment
  • Vulnerability assessment
  • Anti-virus
  • Applications
  • DBs
  • Physical infrastructure
• Establish prioritized log management policies & procedures

Log Management Checklist

1. Scalable architecture
2. Minimal footprint at remote sites
3. Transaction assurance
4. Audit and litigation quality data
5. Universal event collection
6. Ease of manageability
7. ….

Several months ago we evaluated a bunch of log management solutions with several goals in mind. We wanted a solution that was agile enough to be able to take in a wide variety of log formats as well as configuration files. It needed to shield sensitive information (passwords, credit card information, etc) from unauthorized users. It needed to provide us with a customizable interface where we could report on all of the log data it gathered. Lastly, it needed to provide our customers (developers) with the ability to self-service their own log files. After evaluating most of the major players in the log management arena, we found our ideal solution in a product called Splunk.

The first thing I noticed when evaluating Splunk was that they're not like everyone else. They're not trying to sell you some sort of logging appliance and they offer their software free for customers with 100 MB/day or less worth of logging. Getting Splunk installed was a breeze. You can have it up and running in minutes. It truly is Log Management for Dummies in that respect, but under the hood there is a highly configurable and customizable tool with an API that you could use to write your own applications to examine log files.

At this point I've mucked around with Splunk for a few months and our configuration is pretty intense. I've added in custom indexes to make my custom dashboards load faster. I've set Splunk up to create queryable metadata fields based on information in the logs. I've added filters for custom timestamps and auditing so we can tell if a log file has been modified. I've even set up a "deployment server" to distribute Splunk's configuration bundles to my various types of servers. This brings me to the one drawback of Splunk: Upgrading. Rumor has it that they are working on making it easier to upgrade from one version to the next, but for the time being it involves logging in to each server, stopping Splunk, upgrading the files, and restarting Splunk again. If you only had to upgrade every once in a while it would be fine, but they maintain a very active development team so I find myself constantly wanting to upgrade to get the latest bug fixes and features.

Other than that, Splunk does exactly what I tell it to do. It grabs all of our logs and presents them in a single intuitive interface. Think of it as a search engine for log and configuration files. Then, once I have the log data in front of me, I can create custom reports based on that data. If I want to, I can even alert based on information Splunk finds in my logs (send an e-mail to a developer every time their application throws an error message). Oh, did I mention that Splunk has a PCI Dashboard that you can install for free? Ask those other guys how much they charge for their PCI solution.

The next time you have some free time be sure to download Splunk and install it on one of your development servers. You won't be disappointed.