Virtualization Security Best Practices from a Customer’s and Vendor’s Perspective
The next session during the ISSA half-day seminar on Virtualization and Cloud Computing Security was on security best practices from a customer and vendor perspective. It featured Brian Engle, CIO of Temple Inland, and Rob Randell, CISSP and Senior Security Specialist at VMware, Inc. My notes from the presentation are below:
Temple Inland Implementation - Stage 1
Overcome Hurdles
- Management skeptical of Windows virtualization
Don't Fear the Virtual World
- First year:
- Built out development only environment
- Trained staff
- Developed support processes
- Showed hard dollar savings
Temple Inland - Stage 2
- Build QA environment
- Improve processes
- Develop rapid provisioning
- Demonstrate advanced functions
- Vmotion
- P2V Conversions
Temple Inland - Stage 3
First production environment
Temple-Inland Implementation
- Prior to VMWare. Typical remote facility
- Physical domain controller
- Physical application/file server
- Physical tape drive
- New architecture
- Single VMWare server
- No tape drive
- Desktops
- Virtualize desktops through VMWare
- No application issues like Citrix Metaframe
- Quick deployment and repair
How Virtualization Affects Datacenter Security
- Abstraction and Consolidation
- +Capital and Operational Cost Savings
- -New infrastructure layer to be secured
- -Greater impact of attack or misconfiguration
- Collapse of Switches and servers into one device
- +Flexibility
- +Cost-savings
- -Lack of virtual network visibility
- -No separation-by-default of administration
Temple-Inland split the teams so that there was a virtual network administration team within the server administration team.
How Virtualization Affects Datacenter Security
- Faster deployment of servers
- + IT responsiveness
- -Lack of adequate planning
- -Incomplete knowledge of current state of infrastructure
- VM Mobility
- +Improved Service Levels
- -Identity divorced from physical location
- VM Encapsulation
- +Ease of business continuity
- +Consistency of deployment
- +Hardware Independence
- -Outdated offline systems
Build anti-virus, client firewalls, etc into the offline images so that servers are up-to-date right when they are installed.
If something happens to a system, you can't just pull the plug anymore. You have to have policies and processes in place.
With virtualization you can have a true "gold image" instead of having different images for all of the different types of hardware.
Security Advantages of Virtualization
- Allows automation of many manual error prone processes
- Cleaner and easier disaster recovery/business continuity
- Better forensics capabilities
- Faster recovery after an attack
- Patching is safer and more effective
- Better control over desktop resources
- More cost effective security devices
- App virtualization allows de-privileging of end users
- Better lifecycle controls
- Future: Security through VM Introspection
Gartner: "Like their physical counterparts, most security vulnerabilities will be introduced through misconfiguration"
What Not to Worry About
- Hypervisor Attacks
- ALL theoretical, highly complex attacks
- Widely recognized by security community as being only of academic interest
- Irrelevant Architectures
- Apply only to hosted architecture (ie. Workstation) not bare-metal (ie. ESX)
- Hosted architecture generally suitable only when you can trust the guest VM
- Contrived Scenarios
- Involved exploits where best practices around hardening, lockdown, desgin, for virtualization etc not followed or
- Poor general IT infrastructure security is assumed
Are there any Hypervisor Attack Vectors?
There are currently no known hypervisor attack vectors to date that have lead to "VM Escape"
- Architecture Vulnerability
- Designed specifically with isolation in mind
- Software Vulnerability - Possible like with any code written by humans
- Mitigating Circumstances:
- Small Code Footprint of Hypervisor (~21MB) is easier to audit
- If a software vulnerability is found, exploit difficulty will be very high
- Purpose build for virtualization only
- Non-interactive environment
- Less code for hackers to leverage
- Ultimately depends on VMWare security response and patching
- Mitigating Circumstances:
Concern: Virtualizing the DMZ/Mixing Trust Zones
Three Primary Configurations
- Physical separation of trust zones
- Virtual separation of trust zones with physical security devices
- Fully collapsing all servers and security devices into a VI3 infrastructure
Also applies to PCI requirement
Physical Separation of Trust Zones
Advantages
- Simpler, less complex configuration
- Less change to physical environment
- Little change to separation of duties
- Less change in staff knowledge requirements
- Smaller chance of misconfiguration
Disadvantages
- Lower consolidation and utilization of resources
- Higher cost
Virtual Separation of Trust Zones with Physical Security Devices
Advantages
- Better utilization of resources
- Take full advantage of virtualization benefits
- Lower cost
Disadvantages (can be mitigated)
- More complexity
- Greater chance of misconfiguration
Getting more toward "the cloud" where web zone, app zone, and DB zone are all virtualized on the same system, but still using physical firewalls.
Fully Collapsed Trust Zones Including Security Devices
Advantages
- Full utilization of resources, replacing physical security devices with virtual
- Lowest-cost option
- Management of entire DMZ and network from a single management workstation
Disadvantages
- Greatest complexity, which in turn creates highest chance of misconfiguration
- Requirement for explicit configuration to define separation of duties to help mitigate risk of misconfiguration; also requires regualar audits of configurations
- Potential loss of certain functionality, such as VMotion (being mitigated by vendors and VMsafe)
How do we secure our Virtual Infrastructure?
Use the principles of Information Security
- Hardening and lockdown
- Defense in depth
- Authorization, authentication, and accounting
- Separation of duties and least privileges
- Administrative controls
Protect your management interfaces (VCenter)! They are the keys to the kingdom.
Fundamental Design Principles
- Isolate all management networks
- Disable all unneeded services
- Tightly regualte all administrative access
Summary
- Define requirements and ensure vendor/product can deliver
- Consider culture, capability, maturity, architecture and security needs
- Implement under controlled conditions using a defined methodology
- Use the opportunity to improve control deficiencies in existing physical server areas if possible
- Implement processes for review and validation of controls to prevent the introduction of weaknesses
- Round corners where your control environment allows
- Sustain sound practices that maintain required controls
- Leverage the technology to achieve efficiency and improve scale
Beware The Wolf In Supplier’s Clothing
As you all know, the economic climate of 2009 is a cold, cold winter indeed. And like wolves starved by the cold and hardship of the season, our suppliers have turned feral.
When everyone's sales slip due to the down economy, companies (and individual sales reps) are desperate to make their numbers. How are they doing it? By trying to jack up maintenance costs, in some cases by more than 100%! It's way more than isolated incidents; all our maintenance renewals coming up are meeting with hugely inflated quotes. And not fly-by-night companies either, I don't want to name names but let's just say I am confident everyone out there has heard of all of them.
So protect yourself. In your dealings with your supplier reps, start making it clear way ahead of time that your economic situation sucks too and you certainly expect that there's a price freeze in place. Don't put up with it either - they know they're going to make plenty of money off all the goons they send quotes to who will just rubber-stamp it and send it on so they can return to ESPNZone (I'm looking at you, State of Texas). If you put up enough resistance they'll go looking for easier pickings, just like those mean ol' wolves do. We had one outfit that wanted to jack up our maintenance cost by $125k a year, but luckily our IT director is a firm lady who has no problems with browbeating a sales rep until he cries. In the end, we let them have a 5% increase because we ended up feeling sorry for them.
And have a backup plan. If they really do have you over a barrel, then you're low on leverage - you can try offering reference calls, presenting at conferences, and other handy non-cash incentives to them. But when it comes down to it, you need to be able to walk away from them. And to do this you need to plan ahead. There are very few things that there's only one of. Have multiple suppliers lined up, and have a plan to change hardware or software if you have to. Also look into open source, or third party support - even if it's "not as good," these days you have to decide how much good is worth how much money.
Now don't get me wrong, we like to partner with our suppliers and treat them friendly. Win-win and all that. But good fences build good neighbors, and there's nothing friendly about showing up and saying "Hey, your operations will grind to a halt without our product, so stick 'em up and give me double this year!"
Be advised, that gleam in Bob the Sales Rep's eyes will be a little hungrier than usual these days, and he's gotta eat one of God's little forest creatures to live. Just make sure it's not you.