{"id":334,"date":"2009-11-13T10:35:08","date_gmt":"2009-11-13T15:35:08","guid":{"rendered":"http:\/\/www.webadminblog.com\/?p=334"},"modified":"2010-02-01T19:08:24","modified_gmt":"2010-02-02T00:08:24","slug":"application-security-metrics-from-the-organization-on-down-to-the-vulnerabilities","status":"publish","type":"post","link":"https:\/\/www.webadminblog.com\/index.php\/2009\/11\/13\/application-security-metrics-from-the-organization-on-down-to-the-vulnerabilities\/","title":{"rendered":"Application Security Metrics from the Organization on Down to the Vulnerabilities"},"content":{"rendered":"

This presentation was by Chris Wysopal, the CTO of Veracode.\u00a0 My notes are below:<\/p>\n

“To measure is to know.” – James Clerk Maxwell<\/p>\n

“Measurement motivates.” – John Kenneth Galbraith<\/p>\n

Metrics do Matter<\/strong><\/span><\/p>\n

    \n
  1. Metrics quantify the otherwise unquantifiable<\/li>\n
  2. Metrics can show trends and trends matter more than measurements do<\/li>\n
  3. Metrics can show if we are doing a good job or bad job<\/li>\n
  4. Metrics can show if you have no idea where you are<\/li>\n
  5. Metrics establish where “You are here” really is<\/li>\n
  6. Metrics build bridges to managers<\/li>\n
  7. Metrics allow cross sectional comparisons<\/li>\n
  8. Metrics set targets<\/li>\n
  9. Metrics benchmark yourself against the opposition<\/li>\n
  10. Metrics create curiosity<\/li>\n<\/ol>\n

    Metrics Don’t Matter (Mike Rothman)
    \n<\/strong><\/span><\/p>\n

      \n
    • It is too easy to count things for no purpose other than to count them<\/li>\n
    • You cannot measure security so stop<\/li>\n
    • This following is all that matters and you can’t map security metrics to them:\n
        \n
      • Maintenance of availability<\/li>\n
      • Preservation of wealth<\/li>\n
      • Limitation on corporate liability<\/li>\n
      • Compliance<\/li>\n
      • Shepherding the corporate brand<\/li>\n<\/ul>\n<\/li>\n
      • Cost of measurement not worth the benefit<\/li>\n<\/ul>\n

        Bad metrics are worse than no metrics<\/p>\n

        Security Metrics Can Drive Executive Decision Making<\/strong><\/span><\/p>\n

          \n
        • How secure am I?<\/li>\n
        • Am I better off than this time last year?<\/li>\n
        • Am I spending the right about of money?<\/li>\n
        • How do I compare to my peers?<\/li>\n
        • What risk transfer options to I have?<\/li>\n<\/ul>\n

          Goals of Application Security Metrics<\/strong><\/span><\/p>\n

            \n
          • Provide quantifiable information to support enterprise risk management and risk-based decision making<\/li>\n
          • Articulate progress towards goals and objectives<\/li>\n
          • Provides a repeatable, quantifiable way to assess, compare, and track improvements in assurance<\/li>\n
          • Focus activities on risk mitigation in order of priority and exploitability<\/li>\n
          • Facilitate adoption and improvement of secure software design and development processes<\/li>\n
          • Provide and objective means of comparing and benchmarking projects, divisions, organizations, and vendor products<\/li>\n<\/ul>\n

            <\/p>\n

            Use Enumerations<\/strong><\/span><\/p>\n

              \n
            • Enumerations help identify specific software-related items that can be counted, aggregated, evaluated over time<\/li>\n
            • CVE – Common Vulnerabilities and Exposures<\/li>\n
            • CWE – Common Weakness Enumeration<\/li>\n
            • CAPEC – Common Attack Pattern Enumeration and Classification<\/li>\n<\/ul>\n

              Organizational Metrics<\/strong><\/span><\/p>\n

                \n
              • Percentage of application inventory developed with SDLC (which version of SDLC?)<\/li>\n
              • Business criticality of each application in inventory<\/li>\n
              • Percentage of application inventory tested for security (what level of testing?)<\/li>\n
              • Percentage of application inventory remediated and meeting assurance requirements<\/li>\n
              • Roll up of testing results<\/li>\n<\/ul>\n

                Organizational Metrics<\/strong><\/span><\/p>\n

                  \n
                • Cost to fix defects at different points in the software lifecycle<\/li>\n
                • Cost of data breaches related to software vulnerabilities<\/li>\n<\/ul>\n

                  Testing Metrics<\/strong><\/span><\/p>\n

                    \n
                  • Number of threats identified in threat model<\/li>\n
                  • Size of attack surface identified<\/li>\n
                  • Percentage code coverage (static and dynamic)<\/li>\n
                  • Coverage of defect categories (CWE)<\/li>\n
                  • Coverage of attack pattern categories (CAPEC)<\/li>\n<\/ul>\n

                    SANS Top 25 Mapped to Application Security Methods (CWE, Title, Education?, Manual Process?, Tools?, Threat Model?)<\/p>\n

                    Weakness Class Prevalence based on 2008 CVE data (Mitre?)<\/p>\n

                    Basic Metrics: Defect Counts<\/strong><\/span><\/p>\n

                      \n
                    • Design and implementation defects\n
                        \n
                      • CWE identifier<\/li>\n
                      • CVSS score<\/li>\n
                      • Severity<\/li>\n
                      • Likelihood of exploit<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                        Automated Code Analysis Techniques<\/strong><\/span><\/p>\n

                          \n
                        • Static Analysis (White Box Testing)<\/li>\n
                        • Dynamic Analysis (Black Box Testing)<\/li>\n<\/ul>\n

                          Manual Analysis<\/strong><\/span><\/p>\n

                            \n
                          • Manual Penetration Testing<\/li>\n
                          • Manual Code Review<\/li>\n
                          • Manual Design Review<\/li>\n
                          • Threat Modeling<\/li>\n<\/ul>\n

                            WASC Web Application Security Statistics Project 2008<\/strong><\/span><\/p>\n

                              \n
                            • Goals\n
                                \n
                              • Identify the prevalence and probability of different vulnerability classes<\/li>\n
                              • Compare testing methodologies against what types of vulnerabilities they are likely to identify<\/li>\n<\/ul>\n<\/li>\n
                              • Summary\n
                                  \n
                                • 12186 web applications with 97554 detected vulnerabilities<\/li>\n
                                • More than 13% of all reviewed sites can be compromised completely automatically<\/li>\n
                                • About 49% of web applications contain vulnerabilities of high risk level detected by scanning<\/li>\n
                                • Manual and automated assessment by white box method allows to detect these high risk level vulnerabilities with the probability up to 80-96%<\/li>\n
                                • 99% of web applications are not compliant with PCI DSS standard<\/li>\n<\/ul>\n<\/li>\n
                                • Compare to 2007 WASC Project\n
                                    \n
                                  • Number of sites with SQL Injection fell by 13%<\/li>\n
                                  • Number of sites with Cross-site Scripting fell 20%<\/li>\n
                                  • Number of sites with different types of Information Leakage rose by 24%<\/li>\n
                                  • Probability to compromise a host automatically rose from 7 to 13%<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"

                                    This presentation was by Chris Wysopal, the CTO of Veracode.\u00a0 My notes are below: “To measure is to know.” – James Clerk Maxwell “Measurement motivates.” – John Kenneth Galbraith Metrics do Matter Metrics quantify the otherwise unquantifiable Metrics can show trends and trends matter more than measurements do Metrics can show if we are doing […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[343,307],"tags":[76,148,310,344,622,10],"class_list":["post-334","post","type-post","status-publish","format-standard","hentry","category-metrics-security","category-owasp-appsec-dc-2009","tag-application","tag-attack","tag-metrics","tag-probability","tag-security","tag-vulnerability"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pfI0c-5o","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/334","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=334"}],"version-history":[{"count":4,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/334\/revisions"}],"predecessor-version":[{"id":379,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/posts\/334\/revisions\/379"}],"wp:attachment":[{"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=334"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=334"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webadminblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=334"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}