Web Admin Blog Real Web Admins. Real World Experience.


The Importance of Log Management in Today’s Insecure World

For my last session of the first day of the TRISC 2009 Conference, I made the mistake of attending Ricky Allen and Randy Holloway's presentation on "The Importance of Log Management in Today's Insecure World".  I say "mistake" because out of all of the presentations I attended over the entire three days of the conference this was by far the most vendory, the least security oriented, and the worst presentation.  Both of these guys work for ArcSight and while they certainly know their log managment, it was just a lame excuse for a presentation and if I was able to go back in time I would have attended Chip Meadows' presentation on "Pocket protectors, Purple hair and Paranoia" instead as I heard he did a fantastic job.  Anyway, my notes from this presentation are below and the actual slides can be found here:

What is log management?

  • Ensuring your enterprise log data is accessible, easily retrievable and forensically sound
  • Properly dealing with mammoth amounts of event data stores in thousands of vendor generated log files
  • Achieving compliance (SOX, HIPAA, PCI, FISMA), Security and IT operation usage of log data that does not break the bank
  • Log data now represents over 30% of ALL data generated by enterprises – creating a real need for log management
  • Dominant uses for log data include:
    • IT operations – systems/network health and availability
    • Security monitoring – perimeter or insider threat detection
    • Compliance monitoring – for regulations and industry standards

Why should I care?

  • Overwhelming flood of logs
  • Islands of defense
  • Week long manual investigations
  • Massive false positives
  • Heterogeneous consoles
  • Many different formats
  • Regulations and their commonly used frameworks impose various requirements when it comes to log management
  • Regulatory mandates have further increased log retention requirements
  • Increased need to store both security and non-security
  • There continues to be an increased emphasis on audit quality data collection
  • Regulatory requirements
    • SOX: 7yrs
    • PCI: 1yr
    • GLBA: 6yrs
    • EU DR Directive: 2yrs
    • Basel II: 7yrs
    • HIPAA: 6/7yrs
  • Compliance requirements
    • More logging
    • More types of devices
    • Higher volumes of log data
    • Extensive reporting requirements
    • Broader user access
    • Long term retention requirements
    • Audit quality data

What can effective log management do for me?

  • Self-managing & scalable
  • Automated & cost-effective audits
  • IT Operations SLA Efficiency
  • Compliance
  • Simplified Forensic Investigations

Best Practices – NIST 800-92

  • Common log management problems
    • Poor tools and training for staff
    • Laborious and boring
    • Reactive analysis reduces the value of logs
    • Slow response
  • Solutions
    • Establish log management policies & procedures
    • Prioritize log management appropriately
    • Create and maintain a secure log management infrastructure
    • Provide proper support for all staff with log management responsibilities
    • Establish standard log management processes for system-level admins
  • The directive to only log and analyze data this is of the greatest importance helps provide sanity to the logging process
  • Collecting and storing all data regardless of its usefulness increases complexity and deployment costs
  • Secure storage and transmission guideline directly points to the importance of secure and robust capture, transmission and storage of logs
  • Organizations should carefully review the collection architecture, transmission security and access control capabilities of SEM solutions to ensure support of this section of the standard
  • Filtering and aggregation are recommended as a means to only capture logs of security and compliance value based on the corporate retention policy
  • Guideline helps organizations support a “reasonableness” position in not collecting useless log data

Developing a Log Management Program

  • Understand your log management needs (regulatory and operational requirements)
  • Review NIST 800-92
  • Understand your environment
    • Lots devices to collect logs from
    • Multiple locations with no IT staff
    • Collection agents are not an option
    • Network time settings
    • Low bandwith links
  • Devices
    • Firewalls/VPN
    • IDS/IPS
    • Servers and desktop OS
    • Network equipment
    • Vulnerability assessment
    • Anti-virus
    • Applications
    • DBs
    • Physical infrastructure
  • Establish prioritized log management policies & procedures

Log Management Checklist

  1. Scalable architecture
  2. Minimal footprint at remote sites
  3. Transaction assurance
  4. Audit and litigation quality data
  5. Universal event collection
  6. Ease of manageability
  7. ….
Tagged as: , No Comments

Log Management for Dummies (aka Splunk)

Logs are one thing that I think are severely underutilized by most systems administrators. Most of us have taken the first step by actually logging the data, but neglect organizing it into any sort of manageable form. You'll probably argue that any hardcore *nix admin would be able to take the raw logs using grep, cut, awk, and a handful of other *nix power tools and turn it into consumable information, but that'll only get you so far.

Several months ago we evaluated a bunch of log management solutions with several goals in mind. We wanted a solution that was agile enough to be able to take in a wide variety of log formats as well as configuration files. It needed to shield sensitive information (passwords, credit card information, etc) from unauthorized users. It needed to provide us with a customizable interface where we could report on all of the log data it gathered. Lastly, it needed to provide our customers (developers) with the ability to self-service their own log files. After evaluating most of the major players in the log management arena, we found our ideal solution in a product called Splunk.

The first thing I noticed when evaluating Splunk was that they're not like everyone else. They're not trying to sell you some sort of logging appliance and they offer their software free for customers with 100 MB/day or less worth of logging. Getting Splunk installed was a breeze. You can have it up and running in minutes. It truly is Log Management for Dummies in that respect, but under the hood there is a highly configurable and customizable tool with an API that you could use to write your own applications to examine log files.

At this point I've mucked around with Splunk for a few months and our configuration is pretty intense. I've added in custom indexes to make my custom dashboards load faster. I've set Splunk up to create queryable metadata fields based on information in the logs. I've added filters for custom timestamps and auditing so we can tell if a log file has been modified. I've even set up a "deployment server" to distribute Splunk's configuration bundles to my various types of servers. This brings me to the one drawback of Splunk: Upgrading. Rumor has it that they are working on making it easier to upgrade from one version to the next, but for the time being it involves logging in to each server, stopping Splunk, upgrading the files, and restarting Splunk again. If you only had to upgrade every once in a while it would be fine, but they maintain a very active development team so I find myself constantly wanting to upgrade to get the latest bug fixes and features.

Other than that, Splunk does exactly what I tell it to do. It grabs all of our logs and presents them in a single intuitive interface. Think of it as a search engine for log and configuration files. Then, once I have the log data in front of me, I can create custom reports based on that data. If I want to, I can even alert based on information Splunk finds in my logs (send an e-mail to a developer every time their application throws an error message). Oh, did I mention that Splunk has a PCI Dashboard that you can install for free? Ask those other guys how much they charge for their PCI solution.

The next time you have some free time be sure to download Splunk and install it on one of your development servers. You won't be disappointed.