Static Application Vulnerability Testing: Binary Scanning vs Source Code Scanning
I had a meeting yesterday with a vendor who sells a SaaS solution for binary application vulnerability testing. They tell a very interesting story of a world where dynamic testing ("black box") takes place alongside static testing ("white box") to give you a full picture of your application security posture. They even combine the results with some e-Learning aspects so that developers can research the vulnerabilities in the same place they go to find them. In concept, this sounds fantastic, but I quickly turned into a skeptic and as I dug deeper into the details I'm not sure I like what I found.
I wanted to make sure I fully understood what was going on under the hood here so I started asking questions about the static testing and how it works. They've got a nice looking portal where you name your application, give it a version, assign it to a group of developers, and point it to your compiled code (WAR, EAR, JAR, etc). Once you upload your binaries, their system basically runs a disassembler on it to get it into assembly code. It's then at this level that they start looking for vulnerabilities. They said that this process takes about 3 days initially and then maybe 2 days after the first time because they are able to re-use some data about your application. Once complete, they say they are able to provide you a report detailing your vulnerabilities and how to fix them.
The thing that immediately struck me as worth noting here was the 2-3 day turnaround. This means that our developers would need to wait a fairly substantial amount of time before getting any feedback on the vulnerability status of their code. In a world full of Agile development, 2-3 days is a lifetime. Compare that to static source code testing where you get actionable results at compile time. The edge here definitely goes to source code testing as I believe most people would prefer the near-instant gratification.
The next thing worth noting was that they are taking binary files and disassembling them in order to find vulnerabilities. This lends itself to one major issue which is how can you determine with any accuracy the line number of a particular vulnerability written in let's say Java from assembly code generated by disassembling the binaries. By default, it's simply not possible. This vendor claimed that they can by adding in some debug strings at compile time, but even then I'd contend that you're not going to get much. I'm guessing they have some heuristics that are able to tell what function generated a set of assembly code, but I'm extremely skeptical that they can do anything with variable names, custom code functions, etc. I've seen some source code scanners, on the other hand, that not only tell you what line of code is affected, but are able to give you an entire list of parameters that have been consequently affected by that vulnerability. The edge here definitely goes to source code testing.
The main benefit that I can see with binary testing vs source code testing is that we can test code that we didn't write. Things like APIs, third-party applications, open source, etc are all things that we now have visibility into. The only problem here is that while we now can see the vulnerabilities in this software, they are unfortunately all things that we can't directly influence change in, unless we want to send our developers off to work on somebody else's software. I'd argue that scanning for vulnerabilities in that type of code is their responsibility, not ours. Granted, it'd be nice to have validation that there aren't vulnerabilities there that we're exposing ourselves to by uptaking it, but in all honesty are we really going to take the time to scan somebody else's work? Probably not. The edge here goes to binary testing with the caveat being that it's in something that I frankly don't care as much about.
This isn't the complete list of pros and cons by any means. It's just me voicing in writing some concerns that I had about the technology while talking to this particular vendor. In my opinion, the benefits of doing source code testing far outweigh any benefits that we could get from testing compiled binary files. What do you think about the benefits of one versus the other? I'd certainly love for someone to try to change my mind here and show me where the real value lies in binary testing.
Software Assurance Maturity Model (SAMM)
This presentation on the OWASP Software Assurance Maturity Model (SAMM) was by Pravir Chandra, the project lead. I was actually really excited in seeing this topic on the schedule as SAMM is something that I've been toying with for my organization for a while. It's actually a very simple and intuitive approach to how to assess where your organization is at as far as software maturity, where you want to get to, and how to get there. My notes on this presentation are below:
By the end of the presentation should be able to....
- Evaluate an organizations existing software security practices
- Build a balanced software security assurance program in well-defined iterations
- Demonstrate concrete improvements to a security assessment program
- Define and measure security-related activities throughout the organization
Lessons Learned
- Microsoft SDL
- Heavyweight, good for large ISVs
- Touchpoints
- High-level, not enough details to execute against
- CLASP
- Large collection of activities, but no priority ordering
- ALL: Good for experts to use as a guide, but hard for non-security folkds to use off the shelf
Drivers for a Maturity Model
- An organization's behavior changes slowly over time
- Changes must be iterative while working toward long-term goals
- There is no single recipe that works for all organizations
- A solution must enable risk-based choices tailored to the organization
- Guidance related to security activities must be prescriptive
- A solution must provide enough details for non-security-people
- Overall, must be simple, well-defined, and measurable
Therefore, a viable model must...
- Define building blocks for an assurance program
- Delineate all functions within an organization that could be improved over time
- Define how building blocks should be combined
- Make creating change in iterations a no-brainer
SAMM Business Functions (4 in total)
- Start with the core activities tied to any organization performing software development
- Named generically, but should resonate with any developer or manager
- Governance, Construction, Verification, Deployment
SAMM Security Practices (12 in total)
- From each of the Business Functions, 3 Security Practices are defined
- The Security Practices cover all areas relevant to software security assurance
- Each one is a 'silo' for improvement
- Governance: Strategy & Metrics, Education & Guidance, Policy & Compliance
- Construction: Threat Assessment, Security Requirements, Secure Architecture
- Verification: Design Review, Code Review, Security Testing
- Deployment: Vulnerability Management, Environment Hardening, Operational Enablement
What is "software"?
- Lots of different aspects of what software is
- Could be a tarball of source code, UML and specifications, or a server running the code
Under each Security Practice
- Three successive Objectives under each Practice define how it can be improved over time
- Level 1, Level 2, and Level 3
- "Going from crawling to walking to running"
- 72 different actives all about the size of a bread box
Per Level, SAMM defines...
- Objectives
- Activites
- Results
- Success Metrics (2-4 metrics for each objective)
- Costs (training, content, license, or buildout)
- Personnel (overhead on different roles because operating at this level)
Conducting Assessments
- SAMM includes assessment worksheets for each Security Practice
Assessment Process
- Supports both lightweight and detailed assessments
- Organizations may fall in between levels (+)
Creating Scorecards
- Gap Analysis
- Capturing scores from detailed assessments versus expected performance levels
- Demonstrating Improvement
- Capturing scores from before and after an iteration of assurance program buld-out
- Ongoing Measurement
- Capturing scores over consistent tiem frames for an assurance program that is already in place
Roadmap Templates
- To make the "building blocks" usable, SAMM defines Roadmaps templates for typical kinds of organizations
- Independent SW Vendors
- Online Service Providers
- Financial Services Organizations
- Government Organizations
- Organization types chose because
- They represent common use-cases
- Each organization has variations in typical software-induced risk
- Optimal creation of an assurance program is different for each
Expert Contributions
- Build based on collected experiences with 100's of organizations
- Including security experts, developers, architects, development managers, IT managers
Industry Support
- Several case studies already
- Several more case studies underway
The OpenSAMM Project
- http://www.opensamm.org
- Dedicated to defining, improving, and testing the SAMM framework
- Always vendor-neutral, but lots of industry participation
- Targeting new releases every ~18 months
- Change management process
Future Plans
- Mappings to existing standards and regulations
- Additional roadmaps where need is identified
- Additional case studies
Keynote: Collaboratively Advancing Strategies to Mitigate Software Supply Chain Risks
It's my second year at the OWASP AppSec Conference and this year it is in Washington, DC. The New York City Conference last year proved to be probably the best conference I've ever been to. Based on the agenda and the facilities, this year is looking very promising. Today's keynote is by Joe Jarzombeck, the Director for Software Assurance at the National Cyber Security Division for the Office of the Assistant Secretary of Cybersecurity and Communication. Man, is that a mouthful. My notes on the presentation are below:
DHS NCSD Software Assurance Program
- A public/private collaboration that promotes security and software resilence throughout the SDLC
- Reduce exploitable software weaknesses
- Address means to improve capabilities that routinely develop, acquire, and deploy resilent software products
- IT/Software Security risk landscape is a convergence between "defense in depth" and "defense in breadth"
- Applications now cut through the security perimeter
- Rather than attempt to break or defeat network or system security, hackers opt to target application software to circumvent security controls
- 75% of hacks are at the application level
- Most exploitable software vulnerabilities are attributed to non-secure coding practices
- Enable software supply chain transparency
- Acquisition managers and users factored risks posed by software supply chain as part of the trade-space in risk mitigation efforts
- DHS Software Assurance program scoped to address:
- Trustworthiness
- Dependability
- Survivability
- Conformity
- Standalone Common Body of Knowledge (CBK) drawing upon contributing companies/industries
Build Security In: https://buildsecurityin.us-cert.gov
- Focus on making software security a normal part of software engineering
- Process agnostic lifestyle
- There was an interesting slide on touchpoints and artifacts that I took a picture of with my phone and I will try to post here.
Resources to Check Out
"Software Security Engineering: A Guide for Project Managers"
"Enhancing the Development Lifecycle to Produce Secure Software"
Fundamental Practices for Secure Software Development
http://www.safecode.org/publications/SAFECode_Dev_Practices1008.pdf
The Software Assurance Pocket Guide Series
Software Assurance in Acquisition: Mitigating Risks to the Enterprise
- Check out Appendix D - Software Due Diligence Questionnaires
"Making the Business Case for Software Assurance"
"Measuring ... Assurance"
Common Weakness Enumeration (CWE)
- If you have this weakness, then it's not a matter of if, but when you'll be breached.