Web Admin Blog Real Web Admins. Real World Experience.


Why You Shouldn’t Phish Your Users

As an Information Security Program Owner, I get a barrage of e-mails and phone calls multiple times a day from vendors looking to sell us their latest hotness security product.  Between the e-mails, phone calls, expo floor at BlackHat this year, and several talks that I've seen at past conferences, I have noticed a disturbing trend that I thought was worth bringing up: phishing your users.

The concept is simple; you send e-mails to your users with content that appears legitimate along with links or attachments that are designed to simulate a spear-phishing attack.  If the user recognizes it as malicious and deletes it, then they are left to carry on with business as usual.  If, however, they fall victim to your trickery, then they are punished in the form of verbal and written lectures, letters to their management, and security awareness training.  No carrot, all stick.  This situation makes me think back upon an issue that I've encountered with my twin daughters at bedtime.  For over a year we struggled to get them to stay in bed at night.  We would lay them down, play some music, and then leave the room and it wasn't 5 minutes later before they were up playing, yelling, and coming back out into the hall.  We yelled at them, spanked them, turned off lights, and did just about everything we could think of to get them to stay in bed.  None of the punishments we did actually corrected the behavior.  Do you want to know what actually worked?  Offering them a treat in the morning if they stayed in bed all night.  Positive reinforcement.

As much as we hate to admit it, adults aren't that different from children in this way.  Nobody takes well to being tricked into clicking on links or opening attachments.  Punishing them for it leads to even further resentment.  And where do you think they focus those hostilities?  The Security Team.  Those people who you are trying to protect end up blaming you and your team for getting them into trouble.  Now, what happens the next time you have a problem that you need that user's assistance to solve?  Absolutely nothing.  Every time you phish a user, you are burning a bridge that you may need later on.  And since we all know how easy it is to phish a user, it just means that you are burning a lot of bridges.

So, what can we do to prevent our organization from being compromised by phishing and other types of social engineering attacks?  To start with, you should incorporate security awareness training to run alongside your new hire training activities.  Make sure that every employee has a baseline amount of knowledge on the issues and how to avoid them.  Next, you should invest in technologies that will detect and prevent these types of malicious activities.  Performing some sort of link and attachment inspection in e-mails and web content inspection for malware will significantly reduce the success rate of these types of attacks.  Lastly, there are a number of vendors who will track real-life phishing attempts to your users and modify the links to be able to perform analysis on who clicked and who didn't.  This has the exact same effect of phishing your users, where you can sit them down and have a talk about what happened, but without pitting them against the Security Team.  The attacker is now the bad guy and you're just the friendly information security professional helping to get them back up and running and giving them tips so that it doesn't happen again.  You are BUILDING BRIDGES.  And, if you want to put an even more positive spin on this process, offer up a reward for those who get phished, but notify the Security Team instead of clicking on the link or opening the attachment.  Everybody wins.  That's why you shouldn't phish your users.


My First Experiences with a Palo Alto Firewall

I've been following Palo Alto as a networking company for a couple of years now.  Their claim is that the days of the port-based firewall are dead and that their application-centric approach is a far better way to enforce your access controls.  Take the HTTP protocol for example.  HTTP typically runs as a service on port 80, but does that mean that everything running on port 80 is HTTP?  As an attacker looking for a way to funnel data out of your organization, why not use the standard HTTP port to send data, since I know you leave it wide open in order for your employees to surf the web.  There's nothing to say that I actually have to be running an HTTP server on the other end and there's nothing on my classic firewall to tell any differently.  At first, I was admittedly a bit skeptical.  I didn't think that you could really tell enough about different applications on the web to be able to separate them out like Palo Alto claims to.  Fortunately, Palo Alto reached out to me and provided me with a brand new PA-200 in an attempt to change my mind.

When the PA-200 arrived, it came with everything that I would need to get it up and running.  That includes the unit itself, a power supply, a D89 to RJ45 console cable, an ethernet cable, and some instructions and warranty information.


On the front of the unit is four ethernet ports for your devices, a management port, a USB port, a console port, and several status indicator LEDs.


By default, the appliance is configured with ethernet ports 1 and 2 paired as a WAN to LAN link as this is the configuration that the majority of the people who buy it will likely use it for.  That said, by following the instructions to connect your computer up to the management port, you can quickly access the user interface that allows you to change this assignment.

Ethernet Configuration

This shows the ethernet 1 and 2 interfaces as both being a "virtual wire" and here we can see the virtual wire that connects the two.

Virtual Wire

From here, we can take a look at the "zones" and see that our two interfaces have been defined as an untrusted (ethernet 1) and trusted (ethernet 2) zone.


To think of this a different way, my cable modem WAN connection (ie. the Internet) goes in my "untrust" zone and my local network (ie. LAN) goes in my "trust" zone.  Now all that's left is to set our policy and for ease of management to start with, I set it to allow everything out with a default deny all inbound.

Security Profile

With this configuration I had done enough to be up and running on the device and I immediately started to see data populate the dashboard on the top applications running on my network.

Top Applications

It's color coded based on risk level and the dashboard also provides me a similar view of Top High Risk Applications.  Any of these boxes can be clicked on in order to provide additional data about the protocol, sources, destinations, countries, and more.

Application Information

Now, let me say that while I'm running this on my home internet connection, this thing is a hoss and can do way more than I can throw at it.  With their App-ID technology enabled you can throw 100 Mbps of throughput at it no problem.  In addition to being an application firewall, it also does standard port-based firewalling, VPN, routing, switching, and so much more.  It's so extremely versatile that this thing could easily be placed in a smaller branch office and replace multiple other devices on their network such as a firewall, router, and VPN concentrator.  More functionality for less money...who wouldn't want that?  In addition to these default capabilities, additional licensing can also be obtained to allow you to do URL filtering, malware detection, and more.  Having just gotten this up and running, I'm still exploring the ins and outs of all of the functionality, but it's pretty exciting to have all of this capability in a box that is smaller than the cable modem my ISP provides me.  More posts to come on this as I get deeper into the guts of running my new Palo Alto PA-200 !


Combining Tools for Ultimate Malware Threat Intelligence

Last year I gave a talk at a number of different conferences called "The Magic of Symbiotic Security: Creating an Ecosystem of Security Systems" in which I spoke about how if we can break our security tools out of their silos, then they become far more useful.  Lately, I've been doing a lot of work at my company in identifying systems infected by malware and getting rid of the infections because, as you are hopefully aware, the presence of malware on your systems is equivalent to hackers on your network.  Malware can give the controller backdoor access to the system, allows them to scan the network for other devices to compromise, gives them a platform to launch additional attacks from, and enables them to exfiltrate data out of the network.  I have a few different tools which I'll highlight later that do some really cool things on their own, but when you combine their functionality together, you open up a whole new world of possibilities.

The first tool that I wanted to talk about is for malware analysis.  In our case this is FireEye, but this could just as easily be Damballa, Bit9, or any other technology that will allow you to identify IP addresses of hosts infected by malware, servers hosting malware objects, and command and control servers.  Alone, this tool identifies a single client-to-server relationship, but it does provide a pattern that we can use as a template to find similar issues in our environment where perhaps we do not have coverage with this device.  Now that we have identified the patterns that we are looking for, we need to find a way to discover additional instances of those patterns.  This brings me to our second tool.

The second tool is for NetFlow analysis.  In case you are unfamiliar with NetFlow, it is a feature of most network devices that creates summary information about the network activity that is running through them.  It includes the source and destination IP addresses, source and destination ports, protocols, and bytes transferred.  Specifically, we need a NetFlow analysis tool that is capable of showing us connections between our internal systems and systems on the Internet.  In our case, we use a product called LYNXeon to do this.  Alone, LYNXeon does a good job of allowing us to visualize connections from one system to another, but finding the systems related to malware issues can often be a needle in a haystack because of the NetFlow limitations mentioned above.  So while our malware connections (downloads and command-and-control) are buried in the NetFlow data, we really have no way to identify them in the NetFlow tool silo.

Now comes the fun part.  One of the cool things about the FireEye system is that it provides us with the ability to export data and one of the cool things about the LYNXeon system is that it provides us with the ability to import data and tag it.  So what we do is, in FireEye, we export the list of all systems that we have detected as having been infected by malware.  We also export the list of all of the command and control servers and malware hosting servers that we have seen connections to.  Next, we go into LYNXeon and tell it to import these two lists of IP addresses and tag them with a custom tag that we created called "FireEye".  We have now successfully combined these two tools and the payoff is huge.

Success #1: Detecting the Spread of Malware on Your Network

Our FireEye system works by executing downloads inside of a virtual machine and analyzing the affect they have on the system.  Because the virtual machine doesn't always match the target system, in many cases we are only able to tell that it was malware and not that the malware actually infected the system.  Using LYNXeon, however, we can create special queries that will show us all connectivity from the potentially infected system after the time of the malware download.  Did the system immediately make connections to other foreign systems on the Internet?  Did it start scanning our internal network looking for other hosts to compromise?  All this and more is possible now that we have identified a potentially infected system on our network.  Here is a pattern file which I created in LYNXeon to do this:

spreading malware pql query


And here is the pattern diagram which this query accomplishes:

spreading malware pql query diagram

Success #2: Finding Other Infected Systems

FireEye appliances aren't free and with offices in over 40 countries around the world getting full coverage can get expensive.  But, if we can use a handful of appliances to get an idea of where our systems are talking to when compromised, then we have data which we can turn around and use in places where we do not have those appliances.  Because we are sending NetFlow data from our devices around the world into LYNXeon, we can search for any connections to these common malware servers.  No more needle in a haystack.  The data is all there, we just needed to know how to look for it.  Here is a pattern file which I created in LYNXeon to do this:

pql query

And here is the pattern diagram which this query accomplishes:

pql query diagram

Success #3: Discovering Other Types of Attacks

Often times our adversaries aren't just trying one type of attack and giving up when it fails.  They are trying every trick in their arsenal and trying to gain and maintain a foothold on your network with whatever method they can.  Once we've identified an attacker's IP address, we can now use our NetFlow data to see all other traffic coming from that IP address.  Often times, expanding these types of relationships can shed light on other activities they are performing on your network.  Perhaps they are performing reconnaissance on your servers?  Maybe they are trying to DOS one of your systems?  The fact is that once they've been uncovered as a bad guy on your network, you should be weary of all activities performed by them.  Maybe even ban their IP address altogether.  Here is a pattern file which I created in LYNXeon to do this:

other attacks pql query

And here is the pattern diagram which this query accomplishes:

other attacks pql query diagram

So there you have it.  By combining our malware analysis using FireEye and our NetFlow analysis using LYNXeon, we have created a hybrid system capable of far more than either of these tools by themselves.  This is the magic of symbiotic security in action.  Our tools becomes infinitely more powerful when we are able to share the data between them.  Hopefully you will take that into consideration the next time you are looking at purchasing a security tool.


Malware is Using TOR to Bypass Your Domain Blacklists

About a week ago I turned on a new rule on our IPS system that is designed to detect (and block) users who are using TOR to make their activities on our network anonymous.  You can say that TOR is about protecting a user's privacy all you want, but I'd argue that while using corporate assets you should have no expectation of privacy (at least in that sense) and that the use of anonymizers on a corporate network can typically be viewed as a sign that you are up to no good.  Almost immediately when I turned on this new rule, I began seeing associated events in the IPS console.  I decided that the best approach was to contact the user directly as they may be wondering why their Internet connection was no longer working.  I reached out to this particular user and explained that if this was the case, then it was because of the new IPS rule.  The solution was simple; just reconfigure his browser to no longer use TOR as the proxy.  But as I began this process, things started getting weird.

I began by telling the user to look for names like "TOR", "The Onion Router", and "Privoxy" in his Add & Remove Programs.  Strange....there was nothing there.  Then I asked him to check his Task Manager to look for a running process called "tor.exe" or similar.  Again, nothing.  I was at a loss.  I decided that this was something I needed to get my hands on to figure out so I scheduled some time with the user.

This morning when I sat with the user, I noticed little wrong with his system.  He had a few standard applications running, but nothing unusual.  I checked his process listing and saw nothing out of the ordinary.  I ran Hijack This! and that, too, looked pretty normal.  All this, yet in the meantime I continued to see alerts on the IPS system that his computer was using TOR.  Even when I was sitting at the console with NO browser activity.  So, to make a long story short, here's how I finally figured out what was happening.  I checked the IPS system and came up with the source ports for the requests that I was seeing alerts on.  I then went on the system and ran a netstat -nao.  This listed all network connections on the users system along with the associated process.  I checked the list and found the entry that matched the port number I was seeing the alerts on.  I then ran the command tasklist /svc /FI "PID eq <process_num>"  This provided me with the name of the process that was running with this process ID which it turns out was "iexplore.exe".  Wait.  Internet explorer isn't even running on this computer.  Or is it?  Since the default process viewer in the Task Manager is pretty lame, I downloaded the Microsoft Sysinternals Process Monitor.  It's a free tool available from Microsoft and provides a ton more information about running processes and allows you to see what they are doing in real time.  I used the Process Monitor to view these processes and focused particularly on the flags that were used when they started.  What I found was actually pretty startling.

Both of the Internet Explorer processes were started with a special flag that told them to start silently (ie. without the UI) in the background.  They also specified a flag similar to this:

--HiddenServiceDir "C:\Documents and Settings\<User_Name>\Application Data\tor\hidden_service" -- HiddenServicePort "55080"

Aha!  We found our culprit!  TOR was running as a hidden service out of the Application Data directory.  Once I found this, it was all over.  Scanning through the Application Data directory, I also found a file under "Enemvy\ugbie.exe" that was extremely suspect.  A later scan via Malwarebytes identified it as a variant of Trojan.ZbotR.  I deleted these directories and Malwarebytes found one registry key associated with the ugbie.exe file and deleted it.  All is good now and the system is no longer alerting about use of TOR.

So, what's our lesson here?  The malware writers are getting sneaky.  They've realized that we've created blacklists of their servers and they need to be able to adapt around that.  Now, they are using anonymizers, like TOR, to get around these blacklists.  Apparently this isn't the first use of TOR in malware either as I read about something called SkyNet that did something similar.  In any case, they would have gotten away with it if it weren't for my IPS rule to detect TOR and a fair amount of persistence in finding the root cause.  If you're not already detecting this on your network, I think that it's about high time you did it.  You can thank me later.


Visual Correlelation of Security Events

I recently had the opportunity to play with a data analytics platform called LYNXeon by a local company (Austin, TX) called 21CT. The LYNXeon tool is billed as a "Big Data Analytics" tool that can assist you in finding answers among the flood of data that comes from your network and security devices and it does a fantastic job of doing just that. What follows are some of my experiences in using this platform and some of the reasons that I think companies can benefit from the visualizations which it provides.

Where I work, data on security events is in silos all over the place. First, there's the various security event notification systems that my team owns. This consists primarily of our IPS system and our malware prevention system. Next, there are our anti-virus and end-point management systems which are owned by our desktop security team. There's also event and application logs from our various data center systems which are owned by various teams. Lastly, there's our network team who owns the firewalls, the routers, the switches, and the wireless access points. As you can imagine, when trying to reconstruct what happened as part of a security event, the data from each of these systems can play a significant role. Even more important is your ability to correlate the data across these siloed systems to get the complete picture. This is where log management typically comes to play.

Don't get me wrong. I think that log management is great when it comes to correlating the siloed data, but what if you don't know what you're looking for? How do you find a problem that you don't know exists? Enter the LYNXeon platform.

The base of the LYNXeon platform is flow data obtained from your various network device. Regardless of whether you use Juniper JFlow, Cisco NetFlow, or one of the other many flow data options, knowing the data that is going from one place to another is crucial to understanding your network and any events that take place on it. Flow data consists of the following:

  • Source IP address
  • Destination IP address
  • IP protocol
  • Source port
  • Destination port
  • IP type of service

Flow data also can contain information about the size of the data on your network.

The default configuration of LYNXeon basically allows you to visually (and textually) analyze this flow data for issues which is immediately useful.  LYNXeon Analyst Studio comes with a bunch of pre-canned reporting which allows you to quickly sort through your flow data for interesting patterns.  For example, once a system has been compromised, the next step for the attacker is often times data exfiltration.  They want to get as much information out of the company as possible before they are identified and their access is squashed.  LYNXeon provides you with a report to identify the top destinations in terms of data size for outbound connections.  Some other extremely useful reporting that you can do with basic flow data in LYNXeon:

  • Identify DNS queries to non-corporate DNS servers.
  • Identify the use of protocols that are explicitly banned by corporate policy (P2P?  IM?).
  • Find inbound connection attempts from hostile countries.
  • Find outbound connections via internal protocols (SNMP?).

It's not currently part of the default configuration of LYNXeon, but they have some very smart guys working there who can provide services around importing pretty much any data type you can think of into the visualizations as well.  Think about the power of combining the data of what is talking to what along with information about anti-virus alerts, malware alerts, intrusion alerts, and so on.  Now, not only do you know that there was an alert in your IPS system, but you can track every system that target talked with after the fact.  Did it begin scanning the network for other hosts to compromise?  Did it make a call back out to China?  These questions and more can be answered with the visual correlation of events through the LYNXeon platform.  This is something that I have never seen a SIEM or other log management company be able to accomplish.

LYNXeon probably isn't for everybody.  While the interface itself is quite easy to use, it still requires a skilled security professional at the console to be able to analyze the data that is rendered.  And while the built-in analytics help tremendously in finding the proverbial "needle in the haystack", it still takes a trained person to be able to interpret the results.  But if your company has the expertise and the time to go about proactively finding problems, it is definitely worth looking into both from a network troubleshooting (something I really didn't cover) and security event management perspective.