Static Application Vulnerability Testing: Binary Scanning vs Source Code Scanning
I had a meeting yesterday with a vendor who sells a SaaS solution for binary application vulnerability testing. They tell a very interesting story of a world where dynamic testing ("black box") takes place alongside static testing ("white box") to give you a full picture of your application security posture. They even combine the results with some e-Learning aspects so that developers can research the vulnerabilities in the same place they go to find them. In concept, this sounds fantastic, but I quickly turned into a skeptic and as I dug deeper into the details I'm not sure I like what I found.
I wanted to make sure I fully understood what was going on under the hood here so I started asking questions about the static testing and how it works. They've got a nice looking portal where you name your application, give it a version, assign it to a group of developers, and point it to your compiled code (WAR, EAR, JAR, etc). Once you upload your binaries, their system basically runs a disassembler on it to get it into assembly code. It's then at this level that they start looking for vulnerabilities. They said that this process takes about 3 days initially and then maybe 2 days after the first time because they are able to re-use some data about your application. Once complete, they say they are able to provide you a report detailing your vulnerabilities and how to fix them.
The thing that immediately struck me as worth noting here was the 2-3 day turnaround. This means that our developers would need to wait a fairly substantial amount of time before getting any feedback on the vulnerability status of their code. In a world full of Agile development, 2-3 days is a lifetime. Compare that to static source code testing where you get actionable results at compile time. The edge here definitely goes to source code testing as I believe most people would prefer the near-instant gratification.
The next thing worth noting was that they are taking binary files and disassembling them in order to find vulnerabilities. This lends itself to one major issue which is how can you determine with any accuracy the line number of a particular vulnerability written in let's say Java from assembly code generated by disassembling the binaries. By default, it's simply not possible. This vendor claimed that they can by adding in some debug strings at compile time, but even then I'd contend that you're not going to get much. I'm guessing they have some heuristics that are able to tell what function generated a set of assembly code, but I'm extremely skeptical that they can do anything with variable names, custom code functions, etc. I've seen some source code scanners, on the other hand, that not only tell you what line of code is affected, but are able to give you an entire list of parameters that have been consequently affected by that vulnerability. The edge here definitely goes to source code testing.
The main benefit that I can see with binary testing vs source code testing is that we can test code that we didn't write. Things like APIs, third-party applications, open source, etc are all things that we now have visibility into. The only problem here is that while we now can see the vulnerabilities in this software, they are unfortunately all things that we can't directly influence change in, unless we want to send our developers off to work on somebody else's software. I'd argue that scanning for vulnerabilities in that type of code is their responsibility, not ours. Granted, it'd be nice to have validation that there aren't vulnerabilities there that we're exposing ourselves to by uptaking it, but in all honesty are we really going to take the time to scan somebody else's work? Probably not. The edge here goes to binary testing with the caveat being that it's in something that I frankly don't care as much about.
This isn't the complete list of pros and cons by any means. It's just me voicing in writing some concerns that I had about the technology while talking to this particular vendor. In my opinion, the benefits of doing source code testing far outweigh any benefits that we could get from testing compiled binary files. What do you think about the benefits of one versus the other? I'd certainly love for someone to try to change my mind here and show me where the real value lies in binary testing.
Auditors Just Don’t Understand Security
Part of my new role as the Information Security Program Owner at NI is taking care of our regulatory compliance concerns which means I spend quite a bit of time dealing with auditors. Now auditors are nice people and I want to preface what I'll say next by saying that I think auditors do perform a great service to companies. I'm sure that most of them are hard-workers and understand compliance requirements probably better than I do, but they just don't understand security.
As a case in point, we're in the middle of our annual audit by one of those "Big Four" audit firms which I won't name here to protect the innocent. I sent an email checking in with our auditors to make sure that they had everything they needed before we went into our four-day holiday weekend. They said that they had received everything they needed except for documentation on "privileged users from the current OS and Database environments" as well as "evidence of current password settings from the application servers, OS, and Database". We go through a round of translation from Auditorese to Techie and figure out that they want exports of some specific user, profile, role, and privilege tables from the database and copies of /etc/passwd, /etc/shadow, and /etc group from the servers.
So we obtain the requested documentation and I shoot them back an email message to find out their proposed method for transferring the files. Secure FTP? No. PGP encryption? Nope. Their response back was astonishing:
How large do you think they'll be? Email should be fine.
Seriously? These are the guys that we're paying to verify that we're properly protecting our systems and they're suggesting that sending our usernames and password hashes via cleartext email is an appropriate method of file transfer. I respond back:
I'm not really concerned about the size of the files, but rather, the data that they contain. Sending files containing the users, groups, and password hashes for our financial systems via cleartext is probably not a good plan considering the point of this process is protecting that data.
And they respond with:
Whatever you'd like Josh. As long as you have the files as of today, we're good.
So now I'm convinced that auditors (or at least these auditors) view security as nothing more than a checklist. The people telling me what I need to do in order to protect my systems really have no clue about the fundamentals of security. If it's not on their checklist, then it must not be of importance. In this particular situation it may be easier or more convenient to send the documents via email, but any security professional worth their salt would tell you that's not secure nor appropriate for that data. Either our auditors hold themselves to a very different standard than the rest of us security professionals or they just don't understand security unless it's on a checklist.
Looking for DevOps Stuff?
If you heard about us at Velocity 2010 and are coming here for sweet sweet DevOps and agile info, we've moved to a different blog - come see us at the agile admin!
Our First DevOps Implementation
Although we're currently engaged in a more radical agile infrastructure implementation, I thought I'd share our previous evolutionary DevOps implementation here (way before the term was coined, but in retrospect I think it hits a lot of the same notes) and what we learned along the way.
Here at NI we did what I'll call a larval DevOps implementation starting about seven years ago when I came and took over our Web Systems team, essentially an applications administration/operations team for our Web site and other Web-related technologies. There was zero automation and the model was very much "some developers show up with code and we had to put it in production and somehow deal with the many crashes per day resulting from it." We would get 100-200 on-call pages a week from things going wrong in production. We had especially entertaining weeks where Belgian hackers would replace pages on our site with French translations of the Hacker's Manifesto. You know, standard Wild West stuff. You've been there.
Step One: Partner With The Business
First thing I did (remember this is 2002), I partnered with the business leaders to get a "seat at the table" along with the development managers. It turned out that our director of Web marketing was very open to the message of performance, availability, and security and gave us a lot of support.
This is an area where I think we're still ahead of even a lot of the DevOps message. Agile development carries a huge tenet about developers partnering side-by-side with "the business" (end users, domain experts, and whatnot). DevOps is now talking about Ops partnering with developers, but in reality that's a stab at the overall more successful model of "biz, dev, and ops all working together at once."
dev2ops Interview
Want to hear me spout off more about DevOps? Well, here's your chance; I did an interview with Damon Edwards of DTO and they've posted it on the dev2ops blog!
Killer quote:
"I say this as somebody who about 15 years ago chose system administration over development. But system administration and system administrators have allowed themselves to lag in maturity behind what the state of the art is. These new technologies are finally causing us to be held to account to modernize the way we do things. And I think that’s a welcome and healthy challenge."
Amazon Web Services – Convert To/From VMs?
In the recent Amazon AWS Newsletter, they asked the following:
Some customers have asked us about ways to easily convert virtual machines from VMware vSphere, Citrix Xen Server, and Microsoft Hyper-V to Amazon EC2 instances - and vice versa. If this is something that you're interested in, we would like to hear from you. Please send an email to aws-vm@amazon.com describing your needs and use case.
I'll share my reply here for comment!
This is a killer feature that allows a number of important activities.
1. Product VMs. Many suppliers are starting to provide third-party products in the form of VMs instead of software to ease install complexity, or in an attempt to move from a hardware appliance approach to a more-software approach. This pretty much prevents their use in EC2. <cue sad music> As opposed to "Hey, if you can VM-ize your stuff then you're pretty close to being able to offer it as an Amazon AMI or even SaaS offering." <schwing!>
2. Leveraging VM Investments. For any organization that already has a VM infrastructure, it allows for reduction of cost and complexity to be able to manage images in the same way. It also allows for the much promised but under-delivered "cloud bursting" theory where you can run the same systems locally and use Amazon for excess capacity. In the current scheme I could make some AMIs "mostly" like my local VMs - but "close" is not good enough to use in production.
3. Local testing. I'd love to be able to bring my AMIs "down to me" for rapid redeploy. I often find myself having to transfer 2.5 gigs of software up to the cloud, install it, find a problem, have our devs fix it and cut another release, transfer it up again (2 hour wait time again, plus paying $$ for the transfer)...
4. Local troubleshooting. We get an app installed up in the cloud and it's not acting quite right and we need to instrument it somehow to debug. This process is much easier on a local LAN with the developers' PCs with all their stuff installed.
5. Local development. A lot of our development exercises the Amazon APIs. This is one area where Azure has a distinct advantage and can be a threat; in Visual Studio there is a "local Azure fabric" and a dev can write their app and have it running "in Azure" but on their machine, and then when they're ready deploy it up. This is slightly more than VM consumption, it's VMs plus Eucalyptus or similar porting of the Amazon API to the client side, but it's a killer feature.
Xen or VMWare would be fine - frankly this would be big enough for us I'd change virtualization solutions to the one that worked with EC2.
I just asked one of our developers for his take on value for being able to transition between VMs and EC2 to include in this email, and his response is "Well, it's just a no-brainer, right?" Right.
Come to CloudCamp Austin 2!
The second CloudCamp in Austin is happening June 10. It's an unconference about, of course, cloud computing. Read about it and sign up here!
I missed the first one but loved OpsCamp so I'm going!
Simplifying On-call Through Alert Aggregation
One of the coolest things about working on the Web Systems Team at National Instruments is that the company has invested in a wide variety of tools to assist us with our jobs. Since we are responsible for the availability of ni.com, we have the standard URL and content monitors (Sitescope and Nagios). We also have the ability to do real user monitoring with a tool called Coradiant TrueSight. We are also responsible for the website's performance so we have purchased tools like Panorama to diagnose code level issues. We have Splunk for log monitoring and Gomez for a third-party performance and availability monitor. We even have a SaaS provider that does application security scanning. Having all of these tools at our disposal is quite awesome and allows us to quickly find and fix issues with the site. The problem is that every single one of those tools has it's own alerting and reporting interface.
This isn't a new problem by any means. I've seen this issue at every job that I've ever had where the responsibilities included operational support. You rely on multiple tools to tell you when things aren't going quite right, but now you end up spending some non-zero portion of your time managing those tools. For example, lets say that your company has a small release that lasts a few hours once a month. You now have to log in to the control panel (GUI) for each one of those tools and disable your alerts for that time period so that your on-call device isn't going crazy. Assume that you have only four alerting tools and it takes you approximately 5 minutes to log in to each, set the maintenance window, and log back out. You just spent 20 minutes to disable alerts! Now you're getting to the end of the release and things didn't go as planned so the release is running longer than expected. Now you have to spend another 20 minutes to extend the maintenance window. How frustrating is that?
The issue gets even more complicated when you have multiple people providing support in either an on-call rotation or follow-the-sun type of scenario. At NI, we have an operations team that handles alerts during normal business hours, an on-call admin who handles alerts from 5 PM to 2 AM, and then a super-awesome Hungarian Web Admin who takes over responding to pages after 2 AM (9 AM in Hungary). Most of the alerting configurations that these tools provide aren't even able to handle this type of scenario, but let's suppose they did. You're still stuck logging into multiple systems every time there's a holiday, somebody goes on vacation, etc. And what happens if you don't have a dedicated on-call device to pass from person to person? Then you're stuck updating the alert configurations every time the on-call person changes in your rotation.
This really got me thinking that there has to be a better way to do things. I searched the internet looking for a solution, but when I couldn't find anything to do exactly what I wanted it to do, I ended up writing my own. It's now my pleasure to share with you iAlertYou. The idea is quite simple. You take all of those different tools that send alerts and you aggregate them in the same place. In this case, it's on ialertyou.com. By doing this, you gain the ability to control everything from a single, centralized, management platform. Have a maintenance window? No problem. Just log in, set it once, and it affects all of your alerts. Same thing for both alert scheduling (who should get pages and when) and contact groups (used for on-call rotations). Plus, by having all of your alerts going through a single aggregation point, it means that we can also do reporting on all of your alerts. Ever wondered how many of your alerts come from what tools? What times of the day you get the most alerts? It's all possible through alert aggregation.
Certainly there are drawbacks to this type of scenario. Most importantly, you're introducing another dependency in what is typically a mission critical activity. While I can't eliminate this concern completely, I built the system on top of internet cloud technologies for superior scalability. I've architected the application using best-practices in availability, performance, security, and usability. Currently, the only offering is a $30/month "everything" plan, but if you spend more than 10-20 minutes a month changing alert configurations, the ROI is realized very quickly. I will also be rolling out a "free" plan (thanks to Peco) with a limited subset of the functionality. I'd like to invite you to check out http://www.ialertyou.com and see if it can help your company simplify on-call through alert aggregation.
Amazon EC2 EBS Instances and Ephemeral Storage
Here's a couple tidbits I've gleaned that are useful.
When you start an "instance-store" Amazon EC2 instance, you get a certain amount of ephemeral storage allocated and mounted automatically. The amount of space varies by instance size and is defined here. The storage location and format also varies by instance size and is defined here.
The upshot is that if you start an "instance-store" small Linux EC2 instance, it automagically has a free 150 GB /mnt disk and a 1 GB swap partition up and runnin' for ya. (mount points vary by image, but that's where they are in the Amazon Fedora starter.)
[root@domU-12-31-39-00-B2-01 ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 10321208 1636668 8160252 17% / /dev/sda2 153899044 192072 145889348 1% /mnt none 873828 0 873828 0% /dev/shm
[root@domU-12-31-39-00-B2-01 ~]# free total used free shared buffers cached Mem: 1747660 84560 1663100 0 4552 37356 -/+ buffers/cache: 42652 1705008 Swap: 917496 0 917496
But, you say, I am not old or insane! I use EBS-backed images, just as God intended. Well, that's a good point. But when you pull up an EBS image, these ephemeral disk areas are not available to you. The good news is, that's just by default.
The ephemeral storage is still available and can be used (for free!) by an EBS-backed image. You just have to set the block devices up either explicitly when you run the instance or bake them into the image.
Runtime:
You refer to the ephemeral chunks as "ephemeral0", "ephemeral1", etc. - they don't tell you explicitly which is which but basically you just count up based on your instance type (review the doc). For a small image, it has an ephemeral0 (ext3, 15 GB) and an ephemeral1 (swap, 1 GB). To add them to an EBS instance and mount them in the "normal" places, you do:
ec2-run-instances <ami id> -k <your key> --block-device-mapping '/dev/sda2=ephemeral0' --block-device-mapping '/dev/sda3=ephemeral1'
On the instance you have to mount them - add these to /etc/fstab and mount -a or do whatever else it is you like to do:
/dev/sda3 swap swap defaults 0 0 /dev/sda2 /mnt ext3 defaults 0 0
And if you want to turn the swap on immediately, "swapon /dev/sda3".
Image:
You can also bake them into an image. Add a fstab like the one above and when you create the image, do it like this, using the exact same --block-device-mapping flag:
ec2-register -n <ami id> -d "AMI Description" --block-device-mapping /dev/sda2=ephemeral0 --block-device-mapping '/dev/sda3=ephemeral1' --snapshot your-snapname --architecture i386 --kernel<aki id> --ramdisk <ari id>
Ta da. Free storage that doesn't persist. Very useful as /tmp space. Opinion is split among the Linuxerati about whether you want swap space nowadays or not; some people say some mix of "if you're using more than 1.8 GB of RAM you're doing it wrong" and "swapping is horrid, just let bad procs die due to lack of memory and fix them." YMMV.
Ephemeral EBS?
As another helpful tip, let's say you're adding an EBS to an image that you don't want to be persistent when the instance dies. By default, all EBSes are persistent and stick around muddying up your account till you clean them up. If you don't want certain EBS-backed drives to persist, what you do is of the form:
ec2-modify-instance-attribute --block-device-mapping "/dev/sdb=vol-f64c8e9f:true" i-e2a0b08a
Where 'true' means "yes, please, delete me when I'm done." This command throws a stack trace to the tune of
Unexpected error: java.lang.ClassCastException: com.amazon.aes.webservices.client.InstanceBlockDeviceMappingDescription cannot be cast to com.amazon.aes.webservices.client.InstanceBlockDeviceMappingResponseDescription
But it works, that's just a lame API tools bug.
P.S. We’re Hiring
Web Operations Engineer
If you're from the Austin area you probably know National Instruments - our campus is up here at Mopac and Braker; we've been named one of Fortune Magazine's 100 Best Companies To Work For eleven years running.
We've got an opening on a new team building cloud-based Web systems for new SaaS products and Web integration features of our existing hardware and software products.
We need someone to form the core of a new international Web Operations team to provide 24x7 support of these products. You'll work interactively with our R&D engineers, Web programmers, systems engineers, product support organization, and a host of other groups to this end. You'll also help build up the operational environment - monitoring, provisioning, release process, high availability, reporting, performance assurance, security auditing, that kind of thing. You’ll need to be able to document well and train other operations staff.
You can read the "straight" job posting and apply at ni.com/jobs under R&D as "Web Systems Engineer."
Benefits:
- Work with smart people
- Fun, fast-paced environment
- We'll pay you in real money, not WoW gold
- We're growing and profitable ::cough no layoffs cough::
- Room to innovate
You should be skilled in administration of Linux and Windows systems (we run about 50/50), Java and .NET app servers, Web services, search, security, performance, high availability, infrastructure automation, and all the other fun things that Web systems people do.
This is a Web systems administration position with responsibility for realtime operations - it's not Web design, Web applications programming, help desk/end user support, or "back room" systems admin. If you don't want to spend your day configuring load balancing software, figuring out why someone's app is crashing the app server, finding security holes in a Web application, debugging why a new Web page is really slow, getting pages from server monitors, and coding up a great new tool to automate your team's work, this isn't the position for you. There is oncall “pager duty” and some nights/weekends required.
Sorry, no recruiters or H1-B candidates.