13Nov/090
Application Security Metrics from the Organization on Down to the Vulnerabilities
This presentation was by Chris Wysopal, the CTO of Veracode. My notes are below:
"To measure is to know." - James Clerk Maxwell
"Measurement motivates." - John Kenneth Galbraith
Metrics do Matter
- Metrics quantify the otherwise unquantifiable
- Metrics can show trends and trends matter more than measurements do
- Metrics can show if we are doing a good job or bad job
- Metrics can show if you have no idea where you are
- Metrics establish where "You are here" really is
- Metrics build bridges to managers
- Metrics allow cross sectional comparisons
- Metrics set targets
- Metrics benchmark yourself against the opposition
- Metrics create curiosity
Metrics Don't Matter (Mike Rothman)
- It is too easy to count things for no purpose other than to count them
- You cannot measure security so stop
- This following is all that matters and you can't map security metrics to them:
- Maintenance of availability
- Preservation of wealth
- Limitation on corporate liability
- Compliance
- Shepherding the corporate brand
- Cost of measurement not worth the benefit
Bad metrics are worse than no metrics
Security Metrics Can Drive Executive Decision Making
- How secure am I?
- Am I better off than this time last year?
- Am I spending the right about of money?
- How do I compare to my peers?
- What risk transfer options to I have?
Goals of Application Security Metrics
- Provide quantifiable information to support enterprise risk management and risk-based decision making
- Articulate progress towards goals and objectives
- Provides a repeatable, quantifiable way to assess, compare, and track improvements in assurance
- Focus activities on risk mitigation in order of priority and exploitability
- Facilitate adoption and improvement of secure software design and development processes
- Provide and objective means of comparing and benchmarking projects, divisions, organizations, and vendor products
12Nov/090
Enterprise Application Security – GE’s Approach to Solving Root Cause
The first presentation of the day that I went to was by GE's Darren Challey and was about GE's application security program and how he took a holistic approach to securing the enterprise. My notes on this presentation are below:
Why is AppSec so hard?
- AppSec changes rapidly (look at difference between 2004, 2007, and 2010 Top 10)
- Changing landscape
- Increase skill and talen t pool of technically proficient individuals willing to break the law
- Growing volume of financially valuable data online
- Development of criminal markets (black markets) to facilitate conversion to money
- "Attackers now have effective skills, something to steal, and a place to sell it"
- Application Security is a complete one-sided game
- Need to become an enabler (not a barrier)
- Must inject application security earlier through Guidance, Education, and Tools
- Must understand the development and deployment process and integrate rather than mandate
- NIST study on cost to repair defects when found at different stages of software development (http://www.nist.gov/director/prog-ofc/report02-3.pdf)
- Solving the problem of the enterprise (Culture Change)
- Success factors
- Form a mission and strategy
- Develop policy (but not corporate "mandate")
- Gain executive buy-in (cost / benefit / risk)
- Understand the magnitude of problem (metrics)
- Asset inventory and vulnerability management
- Develop standards (what should I do and when?)
- Establish a formal program (strong leadership)
- Focus on education and training materials
- Develop in-house expertise, services and "COE"
- Continuous improvement, measurement, KPI
- Communicate!
- Drive a culture change (shared need, WIIFM)
- Communicate expectations with vendors
- Implement incentives (and penalties)
- Digitize after the process is solid (tools)
- AppSec program mission & structure
- AppSec program strategy
- Policy (guidance) -> Standards (Guidance) -> Training (Education) -> Metrics (tools) -> Security tools (tools) -> Inventory & tracking (tools) -> Monitor & Improve
Guidance
- "GE Application Security Working Group" (Talking to the businesses is critical! Meet every 2 weeks.)
- Secure Coding Guidelines
- Vulnerability Remediation Guide
- Secure Deployment
- Quick Reference Card
- Contractual Language
- Desk Calendars
- Metrics: AppSec calendars helped increase visitors to key Guidance materials (track hits to website docs when certain activities take place)
Education
- CBT1: Intro to AppSec at GE (60 min for any IT person) - why AppSec is important and what happens when you don't do it
- CBT2: GE Best Practices for Secure Coding (90 min)
- CBT3: Attack Profiles & Countermeasures (120 min for security people)
- Developer Awareness Assessment:
- 100's of internally-developed questions
- Randomized questions, timed completion
- Vendors track their own resutls
- Allows tailoring of training/awareness programs
Tools
- - COE AppSec assessment services
- Vendor framework & Metrics
- Compliance handbook
- Common objects repository
- GE Enterprise Application Security
- Scanning and Monitoring tools
- Automation is the way to go (but the tools are not quite there yet)
Metrics
- Measure Vendor AppSec Performance (Avg % Critical/High Vulnerabilities per Assessment vs % Assessments with Zero Critical/High Vulnerabilities)
- Is it making a difference (map avg of critical/high vulnerabilities per assessment)
Forming a Center of Excellence
- Combines the best available people, processes and tools
- Formal training & defined roles (Comprehensive training program for all auditors to ensure skills are kept current and that auditors can provide more than one type of service)
- COE Team structure (tools, research, operations, stakeholder management, queue management, application security auditors
- Application Assessment Types (black/grey box vs white box)
- Application assessment process (map of the workflow with "swim lanes" of who does each step)
- Measure number of vulnerabilities and severities
- Measure customer satisfaction (overall, ease of engagement, responsiveness)