Web Admin Blog Real Web Admins. Real World Experience.

8May/140

Analyzing NetFlow for Data Loss Detection

The 2014 Verizon Data Breach Investigation Report (DBIR) is out and it paints quite the gloomy picture of the world we live in today where cyber security is concerned.  With over 63,000 security incidents and 1,367 confirmed data breaches, the question is no longer if you get popped, but rather, when.  According to the report, data export is second only to credit card theft on the list of threat actions as a result of a breach.  And with the time to compromise typically measured in days and time to discovery measured in weeks or months, Houston, we have a problem.

I've written in the past about all of the cool tricks we've been doing to find malware and other security issues by performing NetFlow analysis using the 21CT LYNXeon tool and this time I've found another trick around data loss detection that I thought was worth writing about.  Before I get into the trick, let's quickly recap NetFlow for those who aren't familiar with it.

Think of NetFlow as the cliff notes of all of the network traffic that your systems handle on a daily basis.  Instead of seeing WHAT data was transmitted (a task for deep packet inspection/DPI), we see the summary of HOW the data was transmitted.  Things like source and destination IP, source and destination port, protocol, and bytes sent and received.  Because many network devices are capable of giving you this information for free, it only makes sense to capture it and start using it for security analytics.

So, now we have our NetFlow and we know that we're going to be breached eventually, the real question becomes how to detect it quickly and remediate before a significant data loss occurs.  Our LYNXeon tool allows us to create patterns of what to look for within NetFlow and other data sources.  So, to help detect for data loss, I've designed the following analytic:

LYNXeon Analytics for Data Loss

What this analytic does is it searches our NetFlow for any time an internal IP address is talking to an external IP address.  Then, it adds up the bytes sent for each of these unique sets of connections (same source, destination, and port) and presents me with a top 25 list.  Something like this:

Top 25 List

So, now we have a list of the top 25 source and destination pairs that are sending data outside of our organization.  There are also some interesting ports in this list like 12547, 22 (SSH), 443 (HTTPS), and 29234.  A system with 38.48 GB worth of data sent to a remote server seems like a bad sign and something that should be investigated.  You get the idea.  It's just a matter of analyzing the data and separating out what is typical vs what isn't and then digging deeper into those.

My advice is to run this report on an automated schedule at least daily so that you can quickly detect when data loss has begun in order to squash it at the source.  You could probably argue that an attacker might take a low and slow approach to remain undetected by my report, and you'd probably be right, but I'd also argue that if this were the case, then I've hopefully slowed them enough to catch them another way within a reasonable timespan.  Remember, security is all about defense in depth and with the many significant issues that are highlighted by the Verizon DBIR, we could use all of the defense we can muster.

20Aug/130

First Impression of LYNXeon 2.29

Let's say that you go to the same restaurant at least once a week for an entire year.  The staff is always friendly, the menu always has something that sounds appealing, and the food is always good enough to keep you coming back for more.  The only real drawback is that it usually takes a solid half-hour to get your food, but you've learned to find something else to do while you're waiting because it's always been worth the wait.  Today you go into the same restaurant, but now the staff goes out of their way to service you, the menu has twice as much selection as before, the food is literally the best thing you've ever tasted, and it was on your table just the way you like it within 30 seconds of placing your order.  This is my initial impression of the newly released version of 21CT's LYNXeon software (version 2.29).

I'll be honest.  Before we upgraded to the new version I had mixed feelings.  On one hand, I loved the data that the LYNXeon platform was giving me.  The ability to comb through NetFlow data and find potentially malicious patterns in it was unlike any other security tool that I've experienced.  On the other hand, the queries sometimes ran for half an hour or more before I had any results to analyze.  I learned to save my queries for when I knew my computer would be sitting idle for a while.  It was a burden that I was willing to undertake for the results, but a burden nonetheless.  We upgraded to LYNXeon 2.29 less than a week ago, but already I can tell that this is a huge leap in the right direction for 21CT's flagship network pattern analysis software.  Those same queries that used to take 30 minutes now take 30 seconds or less to complete.  The reason being is a massive overhaul of the database layer of the platform.  By switching to a grid-based, column-oriented, database structure for storing and querying data, the product was transformed from a pack mule into a thoroughbred.

Enhanced performance wasn't the only feature that found it's way into the 2.29 release.  They also refactored the way that LYNXeon consumes data as well.  While the old platform did a fairly good job of consuming NetFlow data, adding in other data sources to your analytics was a challenge to say the least; usually requiring custom integration work to make it happen.  The new platform has added the concept of a connector with new data types and a framework around how to ingest these different types of data.  It may still require some assistance from support in order to consume data types other than NetFlow, but it's nowhere near the level of effort it was before the upgrade.  We were up and running with the new version of LYNXeon, consuming NetFlow, IPS alerts, and alerts from our FireEye malware prevention system, in a few hours.  The system is capable of adding DNS queries, HTTP queries, and so much more.  What this amounts to is that LYNXeon is now a flexible platform that can allow you to consume data from many different security tools and then visualize and correlate them in one place.  Kinda like a SIEM, but actually useful.

As with any tool, I'm sure that LYNXeon 2.29 won't be without it's share of bugs, but overall the new platform is a huge improvement over the old and with what I've seen so far I gotta say that I'm impressed.  21CT is undoubtedly moving in the right direction and I'm excited to see what these guys do with the platform going forward.  That's my first impression of the 21CT LYNXeon 2.29 release.

7May/130

Combining Tools for Ultimate Malware Threat Intelligence

Last year I gave a talk at a number of different conferences called "The Magic of Symbiotic Security: Creating an Ecosystem of Security Systems" in which I spoke about how if we can break our security tools out of their silos, then they become far more useful.  Lately, I've been doing a lot of work at my company in identifying systems infected by malware and getting rid of the infections because, as you are hopefully aware, the presence of malware on your systems is equivalent to hackers on your network.  Malware can give the controller backdoor access to the system, allows them to scan the network for other devices to compromise, gives them a platform to launch additional attacks from, and enables them to exfiltrate data out of the network.  I have a few different tools which I'll highlight later that do some really cool things on their own, but when you combine their functionality together, you open up a whole new world of possibilities.

The first tool that I wanted to talk about is for malware analysis.  In our case this is FireEye, but this could just as easily be Damballa, Bit9, or any other technology that will allow you to identify IP addresses of hosts infected by malware, servers hosting malware objects, and command and control servers.  Alone, this tool identifies a single client-to-server relationship, but it does provide a pattern that we can use as a template to find similar issues in our environment where perhaps we do not have coverage with this device.  Now that we have identified the patterns that we are looking for, we need to find a way to discover additional instances of those patterns.  This brings me to our second tool.

The second tool is for NetFlow analysis.  In case you are unfamiliar with NetFlow, it is a feature of most network devices that creates summary information about the network activity that is running through them.  It includes the source and destination IP addresses, source and destination ports, protocols, and bytes transferred.  Specifically, we need a NetFlow analysis tool that is capable of showing us connections between our internal systems and systems on the Internet.  In our case, we use a product called LYNXeon to do this.  Alone, LYNXeon does a good job of allowing us to visualize connections from one system to another, but finding the systems related to malware issues can often be a needle in a haystack because of the NetFlow limitations mentioned above.  So while our malware connections (downloads and command-and-control) are buried in the NetFlow data, we really have no way to identify them in the NetFlow tool silo.

Now comes the fun part.  One of the cool things about the FireEye system is that it provides us with the ability to export data and one of the cool things about the LYNXeon system is that it provides us with the ability to import data and tag it.  So what we do is, in FireEye, we export the list of all systems that we have detected as having been infected by malware.  We also export the list of all of the command and control servers and malware hosting servers that we have seen connections to.  Next, we go into LYNXeon and tell it to import these two lists of IP addresses and tag them with a custom tag that we created called "FireEye".  We have now successfully combined these two tools and the payoff is huge.

Success #1: Detecting the Spread of Malware on Your Network

Our FireEye system works by executing downloads inside of a virtual machine and analyzing the affect they have on the system.  Because the virtual machine doesn't always match the target system, in many cases we are only able to tell that it was malware and not that the malware actually infected the system.  Using LYNXeon, however, we can create special queries that will show us all connectivity from the potentially infected system after the time of the malware download.  Did the system immediately make connections to other foreign systems on the Internet?  Did it start scanning our internal network looking for other hosts to compromise?  All this and more is possible now that we have identified a potentially infected system on our network.  Here is a pattern file which I created in LYNXeon to do this:

spreading malware pql query

 

And here is the pattern diagram which this query accomplishes:

spreading malware pql query diagram

Success #2: Finding Other Infected Systems

FireEye appliances aren't free and with offices in over 40 countries around the world getting full coverage can get expensive.  But, if we can use a handful of appliances to get an idea of where our systems are talking to when compromised, then we have data which we can turn around and use in places where we do not have those appliances.  Because we are sending NetFlow data from our devices around the world into LYNXeon, we can search for any connections to these common malware servers.  No more needle in a haystack.  The data is all there, we just needed to know how to look for it.  Here is a pattern file which I created in LYNXeon to do this:

pql query

And here is the pattern diagram which this query accomplishes:

pql query diagram

Success #3: Discovering Other Types of Attacks

Often times our adversaries aren't just trying one type of attack and giving up when it fails.  They are trying every trick in their arsenal and trying to gain and maintain a foothold on your network with whatever method they can.  Once we've identified an attacker's IP address, we can now use our NetFlow data to see all other traffic coming from that IP address.  Often times, expanding these types of relationships can shed light on other activities they are performing on your network.  Perhaps they are performing reconnaissance on your servers?  Maybe they are trying to DOS one of your systems?  The fact is that once they've been uncovered as a bad guy on your network, you should be weary of all activities performed by them.  Maybe even ban their IP address altogether.  Here is a pattern file which I created in LYNXeon to do this:

other attacks pql query

And here is the pattern diagram which this query accomplishes:

other attacks pql query diagram

So there you have it.  By combining our malware analysis using FireEye and our NetFlow analysis using LYNXeon, we have created a hybrid system capable of far more than either of these tools by themselves.  This is the magic of symbiotic security in action.  Our tools becomes infinitely more powerful when we are able to share the data between them.  Hopefully you will take that into consideration the next time you are looking at purchasing a security tool.

12Feb/130

Are Invisible Barbarians At Your Gates?

A couple of weeks back, HD Moore posted a blog entry entitled
"Security Flaws in Universal Plug and Play: Unplug, Don't Play" supporting a Rapid7 Whitepaper in which he discusses the 81 million unique IP addresses that respond to UPnP discovery requests on the Internet and the 23 million fingerprints that match a version of libupnp that exposes the systems to remote code execution.  His research on the subject is fascinating and I highly recommend reading it over, but that's not the reason why I'm writing this.  The first question this research had me asking myself is whether or not my organization utilizes UPnP for anything.  As far as I can tell, the answer to this question is, thankfully, no.  Next, out of curiosity I began to wonder how many people were out there actively trying to find these exploits.  A perfect opportunity to fire up our new LYNXeon tool.

Our LYNXeon tool is configured to consume NetFlow data provided by literally hundreds of routers and switches in our global environment.  One of the most interesting things about it is that it can be used to see the traffic that comes in from our edge routers before it gets squashed by our firewall.  Utilizing this tool in this way, we can visualize the so-called "Barbarians" at our gates.  These are the hackers that are out there trying to find the weak spots in our security in order to get in.  And since I know that UPnP is not a service that we offer up to the Internet at large, it makes finding the guys who are looking to exploit it that much easier.

I fire up LYNXeon and my first step is to generate what is known as "PQL" or "Pattern Query Language".  While their Cyber Analytics Catalog offers up a ton of templates to use to find potential threats, PQL is the base of all those queries and writing your own allows you to define your own catalog of things to look for.  The language is pretty easy to understand.  First you define the characteristics of the connections that you are looking to find.  After doing some research, I found out that HD was looking for openings on UPnP's Simple Service Discovery Protocol (SSDP) service which typically runs on UDP/1900.  So, my query is for connections from external source IPs to internal source IPs using the UDP protocol on port 1900.  Once the connections have been defined, all that is left to do is define the data that you want to see in the results.  In total, my PQL code is 15 lines of code:

ssdp

Now it's officially time to make these invisible Barbarians visible.  I tell LYNXeon to only show me results over the last day (to reduce the amount of time the search takes) and then tell it to "Execute Pattern Search" using the pattern file that I just created.  Searches will vary in time based upon the timeframe searched, the number of forwarding devices, and how complicated your search criteria are.  For me, this search returned 539 results in one minute and 38 seconds.

complete

Now that I have results, I just need to select how to view them.  My personal favorite is viewing the results in the Link Explorer.  This will show my data as nodes on a pictoral graph.  I make one quick adjustment using a organizational feature called "Force Directed Layout" to make the pictures look pretty and voila!

 zoomedout

OK, so zoomed out it looks like a bunch of spider webs.  Now the fun begins as we begin zooming in on each cluster to see what is going on.

 zoomedin

I've blacked out the IP address of the system these guys are connecting to as it is irrelevant for the purposes of this post, but you can clearly see that in the past day this one system has had eight unique IP addresses attempt to connect to it on UDP port 1900.  I've got dozens more just like these on that big graph above with varying degrees of complexity.  From here, LYNXeon allows me to resolve DNS and/or ARIN names for the associated IP addresses.  I can also expand upon those sources to see what else of mine they've been talking to.  Is that cool or what?  It's taken me minutes to find these potential threats and with little more than a few clicks of the mouse.  The Barbarians are most definitely at my gates silently pounding away and chances are pretty good that they are doing the same to you.  The question is....can you find them?

9Oct/120

Visual Correlelation of Security Events

I recently had the opportunity to play with a data analytics platform called LYNXeon by a local company (Austin, TX) called 21CT. The LYNXeon tool is billed as a "Big Data Analytics" tool that can assist you in finding answers among the flood of data that comes from your network and security devices and it does a fantastic job of doing just that. What follows are some of my experiences in using this platform and some of the reasons that I think companies can benefit from the visualizations which it provides.

Where I work, data on security events is in silos all over the place. First, there's the various security event notification systems that my team owns. This consists primarily of our IPS system and our malware prevention system. Next, there are our anti-virus and end-point management systems which are owned by our desktop security team. There's also event and application logs from our various data center systems which are owned by various teams. Lastly, there's our network team who owns the firewalls, the routers, the switches, and the wireless access points. As you can imagine, when trying to reconstruct what happened as part of a security event, the data from each of these systems can play a significant role. Even more important is your ability to correlate the data across these siloed systems to get the complete picture. This is where log management typically comes to play.

Don't get me wrong. I think that log management is great when it comes to correlating the siloed data, but what if you don't know what you're looking for? How do you find a problem that you don't know exists? Enter the LYNXeon platform.

The base of the LYNXeon platform is flow data obtained from your various network device. Regardless of whether you use Juniper JFlow, Cisco NetFlow, or one of the other many flow data options, knowing the data that is going from one place to another is crucial to understanding your network and any events that take place on it. Flow data consists of the following:

  • Source IP address
  • Destination IP address
  • IP protocol
  • Source port
  • Destination port
  • IP type of service

Flow data also can contain information about the size of the data on your network.

The default configuration of LYNXeon basically allows you to visually (and textually) analyze this flow data for issues which is immediately useful.  LYNXeon Analyst Studio comes with a bunch of pre-canned reporting which allows you to quickly sort through your flow data for interesting patterns.  For example, once a system has been compromised, the next step for the attacker is often times data exfiltration.  They want to get as much information out of the company as possible before they are identified and their access is squashed.  LYNXeon provides you with a report to identify the top destinations in terms of data size for outbound connections.  Some other extremely useful reporting that you can do with basic flow data in LYNXeon:

  • Identify DNS queries to non-corporate DNS servers.
  • Identify the use of protocols that are explicitly banned by corporate policy (P2P?  IM?).
  • Find inbound connection attempts from hostile countries.
  • Find outbound connections via internal protocols (SNMP?).

It's not currently part of the default configuration of LYNXeon, but they have some very smart guys working there who can provide services around importing pretty much any data type you can think of into the visualizations as well.  Think about the power of combining the data of what is talking to what along with information about anti-virus alerts, malware alerts, intrusion alerts, and so on.  Now, not only do you know that there was an alert in your IPS system, but you can track every system that target talked with after the fact.  Did it begin scanning the network for other hosts to compromise?  Did it make a call back out to China?  These questions and more can be answered with the visual correlation of events through the LYNXeon platform.  This is something that I have never seen a SIEM or other log management company be able to accomplish.

LYNXeon probably isn't for everybody.  While the interface itself is quite easy to use, it still requires a skilled security professional at the console to be able to analyze the data that is rendered.  And while the built-in analytics help tremendously in finding the proverbial "needle in the haystack", it still takes a trained person to be able to interpret the results.  But if your company has the expertise and the time to go about proactively finding problems, it is definitely worth looking into both from a network troubleshooting (something I really didn't cover) and security event management perspective.