Book Review: Smart & Gets Things Done, by Joel Spolsky
Joel Spolsky is a bit of an internet cause célèbre, the founder of Fog Creek Software and writer of joelonsoftware.com, an influential programming Web site.
The book is about technical recruiting and retention, and even though it’s a small format under 200 page book, it covers a lot of different topics. His focus is on hiring programmers but I think a lot of the same principles apply to hiring for systems admin/Web systems positions. Hiring has been one of the hardest parts of being a Web systems manager, so I got a lot out of the book and tried putting it into practice. Results detailed below!
The Book
The first chapter talks about the relative effectiveness of programmers. We often hire programmers and pay the good ones 10% more than the bad ones. But he has actual data, drawn from a Yale professor who repeatedly teaches the same CS class and assigns the same projects, which shows something that those of us who have been in the field for a long time know – which is that the gap in achievement between the best programmers and the worst ones is a factor of ten. That’s right. In a highly controlled environment, the best programmers completed projects 3-4 times faster than the average and 10x faster than the slowest ones. (And this same relationship holds when adjusting for quality of results.) I’ve been in IT for 15 years and I can guarantee this is true. You can give the same programming task to a bunch of different programmers and get results from “Here, I did it last night” to “Oh, that’ll take three months.” He goes on to note other ways in which you can get 10 mediocre programmers that cannot achieve the same “high notes” as one good programmer. This goes to reinforce how important the programmer, as human capital, is to an organization.
Next, he delves into how you find good developers. Unfortunately, the easy answers don’t work. Posting on monster.com or craigslist gets lots of hits but few keeps. Employee referrals don’t always get the best people either. How do you? He has three suggestions.
- Go to the mountain
- Internships
- Build your own community
“Go to the mountain” means to figure out where the smart people are that you want to hire, and go hang out there. Conferences. Organizations. Web sites. General job sites are zoos, you need venues that are more specifically spot on. Want a security guy? Post on OWASP or ISSA forums, not monster.
We do pretty well with internships, even enhancing that with company sponsored student sourcing/class projects and a large campus recruiting program. He has some good sub-points however – like make your offers early. If you liked them as an intern, offer them a full-time job at that point for when they graduate, don't wait. Waiting puts you into more of a competitive situation. And interns should be paid, given great work to do, and courted for the perm job during the internship.
Building a community – he acknowledges that’s hard. Our company has external communities but not really for IT. For a lot of positions we should be on our our forums like fricking scavengers trying to hire people that post there.
OWASP Top 10 – 2010
This presentation was by Dave WIchers, COO of Aspect Security and an OWASP Board Member. My notes are below:
What's Changed?
- It's about Risks, not just vulnerabilities
- New title is: "The Top 10 Most Critical Web Application Security Risks"
- OWASP Top 10 Risk Rating Methodology
- Based on the OWASP Risk Rating Methodology, used to prioritize Top 10
- 2 Risks Added, 2 Dropped
- Added: A6 - Security Misconfiguration
- Was A10 in 2004 Top 10: Insecure Configuration Management
- Added: A8 - Unvalidated Redirects and Forwards
- Relatively common and VERY dangerous flaw that is not well know
- Removed: A3 - Malicious File Execution
- Primarily a PHP flaw that is dropping in prevalence
- Removed: A6 - Information Leakage and Improper Error Handling
- A very prevalent flaw, that does not introduce much risk (normally)
- Added: A6 - Security Misconfiguration
- A1- Injection: Tricking an application into including unintended commands in the data sent to an interpreter. (http://www.owasp.org/index.php/SQL_Injection_Prevention_Cheat_Sheet)
- A2 - Cross Site Scripting (XSS): Raw data from attacker is sent to an innocent user's browser. For large chunks of user supplied HTML, use OWASP's AntiSamy to sanitize this HTML to make it safe. (http://www.owasp.org/index.php/XSS_(Cross_Site_Scripting)_Prevention_Cheat_Sheet)
- A3 - Broken Authentication and Session Management: Means credentials have to go with every request. Should use SSL for everything requiring authentication.
- A4 - Insecure Direct Object Reference: This is part of enforcing proper "Authorization", along with A7 - Failure to Restrict URL Access.
- A5 - Cross Site Request Forgery (CSRF): An attack where the victim's browser is tricked into issuing a command to a vulnerable web application. Vulnerability is caused by browsers automatically including user authentication data with each request. (Check out OWASP CSRFGuard, OWASP CSRFTester, http://www.owasp.org/index.php/CSRF_Prevention_Cheat_Sheet)
- A6 - Security Misconfiguration: All through the network and platform. Don't forget the development environment. Think of all the places your source code goes. All credentials should change in production.
- A7 - Failure to Restrict URL Access: This is part of enforcing proper "authorization", along with A4 - Insecure Direct Object References.
- A8 - Unvalidated Redirects and Forwards: Web application redirects are very common and frequently include user supplied parameters in the destination URL. If they aren't validated, attacker can send victim to a site of their choice.
- A9 - Insecure Cryptographic Storage: Storing sensitive data insecurely. Failure to identify all sensitive data. Failure to identify all the places that this sensitive data gets stored. Failure to properly protect this data in every location.
- A10 - Insufficient Transport Layer Protection
OWASP Top 10 Risk Rating Methodology
- Attack Vector (How hard for an attacker to use this flaw - 1 (Easy), 2 (Average), 3 (Difficult))
- Weakness Prevalence (How often is it found - 1 (Widespread), 2 (Common), 3 (Uncommon))
- Weakness Detectability (How hard is it for an attacker to find the flaw - 1 (Easy), 2 (Average), 3 (Difficult))
- Technical Impact (1 (Severe), 2 (Moderate), 3 (Minor))
This is generic across the internet, not specific to any organization.
Started a new "Prevention Cheatsheet Series" that the Top 10 references (XSS, SQL Injection, Transport Layer Security, CSRF, Direct Object Reference).
What is actually being released is RC1 of the Top 10 and they are encouraging people to provide comments through the end of the year and then use that feedback to post the final Top 10 in January 2010.
The Importance of Log Management in Today’s Insecure World
For my last session of the first day of the TRISC 2009 Conference, I made the mistake of attending Ricky Allen and Randy Holloway's presentation on "The Importance of Log Management in Today's Insecure World". I say "mistake" because out of all of the presentations I attended over the entire three days of the conference this was by far the most vendory, the least security oriented, and the worst presentation. Both of these guys work for ArcSight and while they certainly know their log managment, it was just a lame excuse for a presentation and if I was able to go back in time I would have attended Chip Meadows' presentation on "Pocket protectors, Purple hair and Paranoia" instead as I heard he did a fantastic job. Anyway, my notes from this presentation are below and the actual slides can be found here:
What is log management?
- Ensuring your enterprise log data is accessible, easily retrievable and forensically sound
- Properly dealing with mammoth amounts of event data stores in thousands of vendor generated log files
- Achieving compliance (SOX, HIPAA, PCI, FISMA), Security and IT operation usage of log data that does not break the bank
- Log data now represents over 30% of ALL data generated by enterprises – creating a real need for log management
- Dominant uses for log data include:
- IT operations – systems/network health and availability
- Security monitoring – perimeter or insider threat detection
- Compliance monitoring – for regulations and industry standards
Why should I care?
- Overwhelming flood of logs
- Islands of defense
- Week long manual investigations
- Massive false positives
- Heterogeneous consoles
- Many different formats
- Regulations and their commonly used frameworks impose various requirements when it comes to log management
- Regulatory mandates have further increased log retention requirements
- Increased need to store both security and non-security
- There continues to be an increased emphasis on audit quality data collection
- Regulatory requirements
- SOX: 7yrs
- PCI: 1yr
- GLBA: 6yrs
- EU DR Directive: 2yrs
- Basel II: 7yrs
- HIPAA: 6/7yrs
- Compliance requirements
- More logging
- More types of devices
- Higher volumes of log data
- Extensive reporting requirements
- Broader user access
- Long term retention requirements
- Audit quality data
What can effective log management do for me?
- Self-managing & scalable
- Automated & cost-effective audits
- IT Operations SLA Efficiency
- Compliance
- Simplified Forensic Investigations
Best Practices – NIST 800-92
- Common log management problems
- Poor tools and training for staff
- Laborious and boring
- Reactive analysis reduces the value of logs
- Slow response
- Solutions
- Establish log management policies & procedures
- Prioritize log management appropriately
- Create and maintain a secure log management infrastructure
- Provide proper support for all staff with log management responsibilities
- Establish standard log management processes for system-level admins
- The directive to only log and analyze data this is of the greatest importance helps provide sanity to the logging process
- Collecting and storing all data regardless of its usefulness increases complexity and deployment costs
- Secure storage and transmission guideline directly points to the importance of secure and robust capture, transmission and storage of logs
- Organizations should carefully review the collection architecture, transmission security and access control capabilities of SEM solutions to ensure support of this section of the standard
- Filtering and aggregation are recommended as a means to only capture logs of security and compliance value based on the corporate retention policy
- Guideline helps organizations support a “reasonableness” position in not collecting useless log data
Developing a Log Management Program
- Understand your log management needs (regulatory and operational requirements)
- Review NIST 800-92
- Understand your environment
- Lots devices to collect logs from
- Multiple locations with no IT staff
- Collection agents are not an option
- Network time settings
- Low bandwith links
- Devices
- Firewalls/VPN
- IDS/IPS
- Servers and desktop OS
- Network equipment
- Vulnerability assessment
- Anti-virus
- Applications
- DBs
- Physical infrastructure
- Establish prioritized log management policies & procedures
Log Management Checklist
- Scalable architecture
- Minimal footprint at remote sites
- Transaction assurance
- Audit and litigation quality data
- Universal event collection
- Ease of manageability
- ….
Log Management for Dummies (aka Splunk)
Logs are one thing that I think are severely underutilized by most systems administrators. Most of us have taken the first step by actually logging the data, but neglect organizing it into any sort of manageable form. You'll probably argue that any hardcore *nix admin would be able to take the raw logs using grep, cut, awk, and a handful of other *nix power tools and turn it into consumable information, but that'll only get you so far.
Several months ago we evaluated a bunch of log management solutions with several goals in mind. We wanted a solution that was agile enough to be able to take in a wide variety of log formats as well as configuration files. It needed to shield sensitive information (passwords, credit card information, etc) from unauthorized users. It needed to provide us with a customizable interface where we could report on all of the log data it gathered. Lastly, it needed to provide our customers (developers) with the ability to self-service their own log files. After evaluating most of the major players in the log management arena, we found our ideal solution in a product called Splunk.
The first thing I noticed when evaluating Splunk was that they're not like everyone else. They're not trying to sell you some sort of logging appliance and they offer their software free for customers with 100 MB/day or less worth of logging. Getting Splunk installed was a breeze. You can have it up and running in minutes. It truly is Log Management for Dummies in that respect, but under the hood there is a highly configurable and customizable tool with an API that you could use to write your own applications to examine log files.
At this point I've mucked around with Splunk for a few months and our configuration is pretty intense. I've added in custom indexes to make my custom dashboards load faster. I've set Splunk up to create queryable metadata fields based on information in the logs. I've added filters for custom timestamps and auditing so we can tell if a log file has been modified. I've even set up a "deployment server" to distribute Splunk's configuration bundles to my various types of servers. This brings me to the one drawback of Splunk: Upgrading. Rumor has it that they are working on making it easier to upgrade from one version to the next, but for the time being it involves logging in to each server, stopping Splunk, upgrading the files, and restarting Splunk again. If you only had to upgrade every once in a while it would be fine, but they maintain a very active development team so I find myself constantly wanting to upgrade to get the latest bug fixes and features.
Other than that, Splunk does exactly what I tell it to do. It grabs all of our logs and presents them in a single intuitive interface. Think of it as a search engine for log and configuration files. Then, once I have the log data in front of me, I can create custom reports based on that data. If I want to, I can even alert based on information Splunk finds in my logs (send an e-mail to a developer every time their application throws an error message). Oh, did I mention that Splunk has a PCI Dashboard that you can install for free? Ask those other guys how much they charge for their PCI solution.
The next time you have some free time be sure to download Splunk and install it on one of your development servers. You won't be disappointed.