The 2014 Verizon Data Breach Investigation Report (DBIR) is out and it paints quite the gloomy picture of the world we live in today where cyber security is concerned. With over 63,000 security incidents and 1,367 confirmed data breaches, the question is no longer if you get popped, but rather, when. According to the report, data export is second only to credit card theft on the list of threat actions as a result of a breach. And with the time to compromise typically measured in days and time to discovery measured in weeks or months, Houston, we have a problem.
I've written in the past about all of the cool tricks we've been doing to find malware and other security issues by performing NetFlow analysis using the 21CT LYNXeon tool and this time I've found another trick around data loss detection that I thought was worth writing about. Before I get into the trick, let's quickly recap NetFlow for those who aren't familiar with it.
Think of NetFlow as the cliff notes of all of the network traffic that your systems handle on a daily basis. Instead of seeing WHAT data was transmitted (a task for deep packet inspection/DPI), we see the summary of HOW the data was transmitted. Things like source and destination IP, source and destination port, protocol, and bytes sent and received. Because many network devices are capable of giving you this information for free, it only makes sense to capture it and start using it for security analytics.
So, now we have our NetFlow and we know that we're going to be breached eventually, the real question becomes how to detect it quickly and remediate before a significant data loss occurs. Our LYNXeon tool allows us to create patterns of what to look for within NetFlow and other data sources. So, to help detect for data loss, I've designed the following analytic:
What this analytic does is it searches our NetFlow for any time an internal IP address is talking to an external IP address. Then, it adds up the bytes sent for each of these unique sets of connections (same source, destination, and port) and presents me with a top 25 list. Something like this:
So, now we have a list of the top 25 source and destination pairs that are sending data outside of our organization. There are also some interesting ports in this list like 12547, 22 (SSH), 443 (HTTPS), and 29234. A system with 38.48 GB worth of data sent to a remote server seems like a bad sign and something that should be investigated. You get the idea. It's just a matter of analyzing the data and separating out what is typical vs what isn't and then digging deeper into those.
My advice is to run this report on an automated schedule at least daily so that you can quickly detect when data loss has begun in order to squash it at the source. You could probably argue that an attacker might take a low and slow approach to remain undetected by my report, and you'd probably be right, but I'd also argue that if this were the case, then I've hopefully slowed them enough to catch them another way within a reasonable timespan. Remember, security is all about defense in depth and with the many significant issues that are highlighted by the Verizon DBIR, we could use all of the defense we can muster.
Let's say that you go to the same restaurant at least once a week for an entire year. The staff is always friendly, the menu always has something that sounds appealing, and the food is always good enough to keep you coming back for more. The only real drawback is that it usually takes a solid half-hour to get your food, but you've learned to find something else to do while you're waiting because it's always been worth the wait. Today you go into the same restaurant, but now the staff goes out of their way to service you, the menu has twice as much selection as before, the food is literally the best thing you've ever tasted, and it was on your table just the way you like it within 30 seconds of placing your order. This is my initial impression of the newly released version of 21CT's LYNXeon software (version 2.29).
I'll be honest. Before we upgraded to the new version I had mixed feelings. On one hand, I loved the data that the LYNXeon platform was giving me. The ability to comb through NetFlow data and find potentially malicious patterns in it was unlike any other security tool that I've experienced. On the other hand, the queries sometimes ran for half an hour or more before I had any results to analyze. I learned to save my queries for when I knew my computer would be sitting idle for a while. It was a burden that I was willing to undertake for the results, but a burden nonetheless. We upgraded to LYNXeon 2.29 less than a week ago, but already I can tell that this is a huge leap in the right direction for 21CT's flagship network pattern analysis software. Those same queries that used to take 30 minutes now take 30 seconds or less to complete. The reason being is a massive overhaul of the database layer of the platform. By switching to a grid-based, column-oriented, database structure for storing and querying data, the product was transformed from a pack mule into a thoroughbred.
Enhanced performance wasn't the only feature that found it's way into the 2.29 release. They also refactored the way that LYNXeon consumes data as well. While the old platform did a fairly good job of consuming NetFlow data, adding in other data sources to your analytics was a challenge to say the least; usually requiring custom integration work to make it happen. The new platform has added the concept of a connector with new data types and a framework around how to ingest these different types of data. It may still require some assistance from support in order to consume data types other than NetFlow, but it's nowhere near the level of effort it was before the upgrade. We were up and running with the new version of LYNXeon, consuming NetFlow, IPS alerts, and alerts from our FireEye malware prevention system, in a few hours. The system is capable of adding DNS queries, HTTP queries, and so much more. What this amounts to is that LYNXeon is now a flexible platform that can allow you to consume data from many different security tools and then visualize and correlate them in one place. Kinda like a SIEM, but actually useful.
As with any tool, I'm sure that LYNXeon 2.29 won't be without it's share of bugs, but overall the new platform is a huge improvement over the old and with what I've seen so far I gotta say that I'm impressed. 21CT is undoubtedly moving in the right direction and I'm excited to see what these guys do with the platform going forward. That's my first impression of the 21CT LYNXeon 2.29 release.
Last year I gave a talk at a number of different conferences called "The Magic of Symbiotic Security: Creating an Ecosystem of Security Systems" in which I spoke about how if we can break our security tools out of their silos, then they become far more useful. Lately, I've been doing a lot of work at my company in identifying systems infected by malware and getting rid of the infections because, as you are hopefully aware, the presence of malware on your systems is equivalent to hackers on your network. Malware can give the controller backdoor access to the system, allows them to scan the network for other devices to compromise, gives them a platform to launch additional attacks from, and enables them to exfiltrate data out of the network. I have a few different tools which I'll highlight later that do some really cool things on their own, but when you combine their functionality together, you open up a whole new world of possibilities.
The first tool that I wanted to talk about is for malware analysis. In our case this is FireEye, but this could just as easily be Damballa, Bit9, or any other technology that will allow you to identify IP addresses of hosts infected by malware, servers hosting malware objects, and command and control servers. Alone, this tool identifies a single client-to-server relationship, but it does provide a pattern that we can use as a template to find similar issues in our environment where perhaps we do not have coverage with this device. Now that we have identified the patterns that we are looking for, we need to find a way to discover additional instances of those patterns. This brings me to our second tool.
The second tool is for NetFlow analysis. In case you are unfamiliar with NetFlow, it is a feature of most network devices that creates summary information about the network activity that is running through them. It includes the source and destination IP addresses, source and destination ports, protocols, and bytes transferred. Specifically, we need a NetFlow analysis tool that is capable of showing us connections between our internal systems and systems on the Internet. In our case, we use a product called LYNXeon to do this. Alone, LYNXeon does a good job of allowing us to visualize connections from one system to another, but finding the systems related to malware issues can often be a needle in a haystack because of the NetFlow limitations mentioned above. So while our malware connections (downloads and command-and-control) are buried in the NetFlow data, we really have no way to identify them in the NetFlow tool silo.
Now comes the fun part. One of the cool things about the FireEye system is that it provides us with the ability to export data and one of the cool things about the LYNXeon system is that it provides us with the ability to import data and tag it. So what we do is, in FireEye, we export the list of all systems that we have detected as having been infected by malware. We also export the list of all of the command and control servers and malware hosting servers that we have seen connections to. Next, we go into LYNXeon and tell it to import these two lists of IP addresses and tag them with a custom tag that we created called "FireEye". We have now successfully combined these two tools and the payoff is huge.
Success #1: Detecting the Spread of Malware on Your Network
Our FireEye system works by executing downloads inside of a virtual machine and analyzing the affect they have on the system. Because the virtual machine doesn't always match the target system, in many cases we are only able to tell that it was malware and not that the malware actually infected the system. Using LYNXeon, however, we can create special queries that will show us all connectivity from the potentially infected system after the time of the malware download. Did the system immediately make connections to other foreign systems on the Internet? Did it start scanning our internal network looking for other hosts to compromise? All this and more is possible now that we have identified a potentially infected system on our network. Here is a pattern file which I created in LYNXeon to do this:
And here is the pattern diagram which this query accomplishes:
Success #2: Finding Other Infected Systems
FireEye appliances aren't free and with offices in over 40 countries around the world getting full coverage can get expensive. But, if we can use a handful of appliances to get an idea of where our systems are talking to when compromised, then we have data which we can turn around and use in places where we do not have those appliances. Because we are sending NetFlow data from our devices around the world into LYNXeon, we can search for any connections to these common malware servers. No more needle in a haystack. The data is all there, we just needed to know how to look for it. Here is a pattern file which I created in LYNXeon to do this:
And here is the pattern diagram which this query accomplishes:
Success #3: Discovering Other Types of Attacks
Often times our adversaries aren't just trying one type of attack and giving up when it fails. They are trying every trick in their arsenal and trying to gain and maintain a foothold on your network with whatever method they can. Once we've identified an attacker's IP address, we can now use our NetFlow data to see all other traffic coming from that IP address. Often times, expanding these types of relationships can shed light on other activities they are performing on your network. Perhaps they are performing reconnaissance on your servers? Maybe they are trying to DOS one of your systems? The fact is that once they've been uncovered as a bad guy on your network, you should be weary of all activities performed by them. Maybe even ban their IP address altogether. Here is a pattern file which I created in LYNXeon to do this:
And here is the pattern diagram which this query accomplishes:
So there you have it. By combining our malware analysis using FireEye and our NetFlow analysis using LYNXeon, we have created a hybrid system capable of far more than either of these tools by themselves. This is the magic of symbiotic security in action. Our tools becomes infinitely more powerful when we are able to share the data between them. Hopefully you will take that into consideration the next time you are looking at purchasing a security tool.
A couple of weeks back, HD Moore posted a blog entry entitled
"Security Flaws in Universal Plug and Play: Unplug, Don't Play" supporting a Rapid7 Whitepaper in which he discusses the 81 million unique IP addresses that respond to UPnP discovery requests on the Internet and the 23 million fingerprints that match a version of libupnp that exposes the systems to remote code execution. His research on the subject is fascinating and I highly recommend reading it over, but that's not the reason why I'm writing this. The first question this research had me asking myself is whether or not my organization utilizes UPnP for anything. As far as I can tell, the answer to this question is, thankfully, no. Next, out of curiosity I began to wonder how many people were out there actively trying to find these exploits. A perfect opportunity to fire up our new LYNXeon tool.
Our LYNXeon tool is configured to consume NetFlow data provided by literally hundreds of routers and switches in our global environment. One of the most interesting things about it is that it can be used to see the traffic that comes in from our edge routers before it gets squashed by our firewall. Utilizing this tool in this way, we can visualize the so-called "Barbarians" at our gates. These are the hackers that are out there trying to find the weak spots in our security in order to get in. And since I know that UPnP is not a service that we offer up to the Internet at large, it makes finding the guys who are looking to exploit it that much easier.
I fire up LYNXeon and my first step is to generate what is known as "PQL" or "Pattern Query Language". While their Cyber Analytics Catalog offers up a ton of templates to use to find potential threats, PQL is the base of all those queries and writing your own allows you to define your own catalog of things to look for. The language is pretty easy to understand. First you define the characteristics of the connections that you are looking to find. After doing some research, I found out that HD was looking for openings on UPnP's Simple Service Discovery Protocol (SSDP) service which typically runs on UDP/1900. So, my query is for connections from external source IPs to internal source IPs using the UDP protocol on port 1900. Once the connections have been defined, all that is left to do is define the data that you want to see in the results. In total, my PQL code is 15 lines of code:
Now it's officially time to make these invisible Barbarians visible. I tell LYNXeon to only show me results over the last day (to reduce the amount of time the search takes) and then tell it to "Execute Pattern Search" using the pattern file that I just created. Searches will vary in time based upon the timeframe searched, the number of forwarding devices, and how complicated your search criteria are. For me, this search returned 539 results in one minute and 38 seconds.
Now that I have results, I just need to select how to view them. My personal favorite is viewing the results in the Link Explorer. This will show my data as nodes on a pictoral graph. I make one quick adjustment using a organizational feature called "Force Directed Layout" to make the pictures look pretty and voila!
OK, so zoomed out it looks like a bunch of spider webs. Now the fun begins as we begin zooming in on each cluster to see what is going on.
I've blacked out the IP address of the system these guys are connecting to as it is irrelevant for the purposes of this post, but you can clearly see that in the past day this one system has had eight unique IP addresses attempt to connect to it on UDP port 1900. I've got dozens more just like these on that big graph above with varying degrees of complexity. From here, LYNXeon allows me to resolve DNS and/or ARIN names for the associated IP addresses. I can also expand upon those sources to see what else of mine they've been talking to. Is that cool or what? It's taken me minutes to find these potential threats and with little more than a few clicks of the mouse. The Barbarians are most definitely at my gates silently pounding away and chances are pretty good that they are doing the same to you. The question is....can you find them?