I've been following Palo Alto as a networking company for a couple of years now. Their claim is that the days of the port-based firewall are dead and that their application-centric approach is a far better way to enforce your access controls. Take the HTTP protocol for example. HTTP typically runs as a service on port 80, but does that mean that everything running on port 80 is HTTP? As an attacker looking for a way to funnel data out of your organization, why not use the standard HTTP port to send data, since I know you leave it wide open in order for your employees to surf the web. There's nothing to say that I actually have to be running an HTTP server on the other end and there's nothing on my classic firewall to tell any differently. At first, I was admittedly a bit skeptical. I didn't think that you could really tell enough about different applications on the web to be able to separate them out like Palo Alto claims to. Fortunately, Palo Alto reached out to me and provided me with a brand new PA-200 in an attempt to change my mind.
When the PA-200 arrived, it came with everything that I would need to get it up and running. That includes the unit itself, a power supply, a D89 to RJ45 console cable, an ethernet cable, and some instructions and warranty information.
On the front of the unit is four ethernet ports for your devices, a management port, a USB port, a console port, and several status indicator LEDs.
By default, the appliance is configured with ethernet ports 1 and 2 paired as a WAN to LAN link as this is the configuration that the majority of the people who buy it will likely use it for. That said, by following the instructions to connect your computer up to the management port, you can quickly access the user interface that allows you to change this assignment.
This shows the ethernet 1 and 2 interfaces as both being a "virtual wire" and here we can see the virtual wire that connects the two.
From here, we can take a look at the "zones" and see that our two interfaces have been defined as an untrusted (ethernet 1) and trusted (ethernet 2) zone.
To think of this a different way, my cable modem WAN connection (ie. the Internet) goes in my "untrust" zone and my local network (ie. LAN) goes in my "trust" zone. Now all that's left is to set our policy and for ease of management to start with, I set it to allow everything out with a default deny all inbound.
With this configuration I had done enough to be up and running on the device and I immediately started to see data populate the dashboard on the top applications running on my network.
It's color coded based on risk level and the dashboard also provides me a similar view of Top High Risk Applications. Any of these boxes can be clicked on in order to provide additional data about the protocol, sources, destinations, countries, and more.
Now, let me say that while I'm running this on my home internet connection, this thing is a hoss and can do way more than I can throw at it. With their App-ID technology enabled you can throw 100 Mbps of throughput at it no problem. In addition to being an application firewall, it also does standard port-based firewalling, VPN, routing, switching, and so much more. It's so extremely versatile that this thing could easily be placed in a smaller branch office and replace multiple other devices on their network such as a firewall, router, and VPN concentrator. More functionality for less money...who wouldn't want that? In addition to these default capabilities, additional licensing can also be obtained to allow you to do URL filtering, malware detection, and more. Having just gotten this up and running, I'm still exploring the ins and outs of all of the functionality, but it's pretty exciting to have all of this capability in a box that is smaller than the cable modem my ISP provides me. More posts to come on this as I get deeper into the guts of running my new Palo Alto PA-200 !
The other day I read that Comcast is launching a new plan to turn home internet users into unwilling participants in their new global wifi strategy. I'm sure that they will soon be touting how insanely awesome it will be to get "full strength" internet access virtually anywhere just by subscribing to this service. Other than the issues with taking a service that the consumer already pays for and carving out their bandwidth for other people, the security practitioner in me can't help but wonder what the security ramifications of sharing an internet connection like this actually means. Combine this with the default access to your cable modem that your service provider already has, and it paints a very scary picture of network security for the home user. It is no longer sufficient (if it ever was) to rely on your cable modem for network access controls. Thus, I am advocating in favor of placing a personal firewall between your cable modem and your network for all home internet setups.
Now, it's not as bad as you may think. It doesn't have to be some crazy expensive piece of equipment like you'd purchase for a business. Even the basic home gateways come with the ability to do Network Address Translation (NAT) which effectively turns your internet connection into a one-way pipe. All I'm saying is that instead of plugging your network devices directly into the cable modem for Internet access, you should use your own hardware and draw a clear "line in the sand" between your equipment and theirs. In addition, I would advocate that you should no longer consider the wifi access provided by the cable modem device as safe and should use your own equipment for this access. In other words, treat anything on the WAN side of your home gateway/personal firewall as untrusted and protect against it accordingly.
The 2014 Verizon Data Breach Investigation Report (DBIR) is out and it paints quite the gloomy picture of the world we live in today where cyber security is concerned. With over 63,000 security incidents and 1,367 confirmed data breaches, the question is no longer if you get popped, but rather, when. According to the report, data export is second only to credit card theft on the list of threat actions as a result of a breach. And with the time to compromise typically measured in days and time to discovery measured in weeks or months, Houston, we have a problem.
I've written in the past about all of the cool tricks we've been doing to find malware and other security issues by performing NetFlow analysis using the 21CT LYNXeon tool and this time I've found another trick around data loss detection that I thought was worth writing about. Before I get into the trick, let's quickly recap NetFlow for those who aren't familiar with it.
Think of NetFlow as the cliff notes of all of the network traffic that your systems handle on a daily basis. Instead of seeing WHAT data was transmitted (a task for deep packet inspection/DPI), we see the summary of HOW the data was transmitted. Things like source and destination IP, source and destination port, protocol, and bytes sent and received. Because many network devices are capable of giving you this information for free, it only makes sense to capture it and start using it for security analytics.
So, now we have our NetFlow and we know that we're going to be breached eventually, the real question becomes how to detect it quickly and remediate before a significant data loss occurs. Our LYNXeon tool allows us to create patterns of what to look for within NetFlow and other data sources. So, to help detect for data loss, I've designed the following analytic:
What this analytic does is it searches our NetFlow for any time an internal IP address is talking to an external IP address. Then, it adds up the bytes sent for each of these unique sets of connections (same source, destination, and port) and presents me with a top 25 list. Something like this:
So, now we have a list of the top 25 source and destination pairs that are sending data outside of our organization. There are also some interesting ports in this list like 12547, 22 (SSH), 443 (HTTPS), and 29234. A system with 38.48 GB worth of data sent to a remote server seems like a bad sign and something that should be investigated. You get the idea. It's just a matter of analyzing the data and separating out what is typical vs what isn't and then digging deeper into those.
My advice is to run this report on an automated schedule at least daily so that you can quickly detect when data loss has begun in order to squash it at the source. You could probably argue that an attacker might take a low and slow approach to remain undetected by my report, and you'd probably be right, but I'd also argue that if this were the case, then I've hopefully slowed them enough to catch them another way within a reasonable timespan. Remember, security is all about defense in depth and with the many significant issues that are highlighted by the Verizon DBIR, we could use all of the defense we can muster.
Let's say that you go to the same restaurant at least once a week for an entire year. The staff is always friendly, the menu always has something that sounds appealing, and the food is always good enough to keep you coming back for more. The only real drawback is that it usually takes a solid half-hour to get your food, but you've learned to find something else to do while you're waiting because it's always been worth the wait. Today you go into the same restaurant, but now the staff goes out of their way to service you, the menu has twice as much selection as before, the food is literally the best thing you've ever tasted, and it was on your table just the way you like it within 30 seconds of placing your order. This is my initial impression of the newly released version of 21CT's LYNXeon software (version 2.29).
I'll be honest. Before we upgraded to the new version I had mixed feelings. On one hand, I loved the data that the LYNXeon platform was giving me. The ability to comb through NetFlow data and find potentially malicious patterns in it was unlike any other security tool that I've experienced. On the other hand, the queries sometimes ran for half an hour or more before I had any results to analyze. I learned to save my queries for when I knew my computer would be sitting idle for a while. It was a burden that I was willing to undertake for the results, but a burden nonetheless. We upgraded to LYNXeon 2.29 less than a week ago, but already I can tell that this is a huge leap in the right direction for 21CT's flagship network pattern analysis software. Those same queries that used to take 30 minutes now take 30 seconds or less to complete. The reason being is a massive overhaul of the database layer of the platform. By switching to a grid-based, column-oriented, database structure for storing and querying data, the product was transformed from a pack mule into a thoroughbred.
Enhanced performance wasn't the only feature that found it's way into the 2.29 release. They also refactored the way that LYNXeon consumes data as well. While the old platform did a fairly good job of consuming NetFlow data, adding in other data sources to your analytics was a challenge to say the least; usually requiring custom integration work to make it happen. The new platform has added the concept of a connector with new data types and a framework around how to ingest these different types of data. It may still require some assistance from support in order to consume data types other than NetFlow, but it's nowhere near the level of effort it was before the upgrade. We were up and running with the new version of LYNXeon, consuming NetFlow, IPS alerts, and alerts from our FireEye malware prevention system, in a few hours. The system is capable of adding DNS queries, HTTP queries, and so much more. What this amounts to is that LYNXeon is now a flexible platform that can allow you to consume data from many different security tools and then visualize and correlate them in one place. Kinda like a SIEM, but actually useful.
As with any tool, I'm sure that LYNXeon 2.29 won't be without it's share of bugs, but overall the new platform is a huge improvement over the old and with what I've seen so far I gotta say that I'm impressed. 21CT is undoubtedly moving in the right direction and I'm excited to see what these guys do with the platform going forward. That's my first impression of the 21CT LYNXeon 2.29 release.
Last year I gave a talk at a number of different conferences called "The Magic of Symbiotic Security: Creating an Ecosystem of Security Systems" in which I spoke about how if we can break our security tools out of their silos, then they become far more useful. Lately, I've been doing a lot of work at my company in identifying systems infected by malware and getting rid of the infections because, as you are hopefully aware, the presence of malware on your systems is equivalent to hackers on your network. Malware can give the controller backdoor access to the system, allows them to scan the network for other devices to compromise, gives them a platform to launch additional attacks from, and enables them to exfiltrate data out of the network. I have a few different tools which I'll highlight later that do some really cool things on their own, but when you combine their functionality together, you open up a whole new world of possibilities.
The first tool that I wanted to talk about is for malware analysis. In our case this is FireEye, but this could just as easily be Damballa, Bit9, or any other technology that will allow you to identify IP addresses of hosts infected by malware, servers hosting malware objects, and command and control servers. Alone, this tool identifies a single client-to-server relationship, but it does provide a pattern that we can use as a template to find similar issues in our environment where perhaps we do not have coverage with this device. Now that we have identified the patterns that we are looking for, we need to find a way to discover additional instances of those patterns. This brings me to our second tool.
The second tool is for NetFlow analysis. In case you are unfamiliar with NetFlow, it is a feature of most network devices that creates summary information about the network activity that is running through them. It includes the source and destination IP addresses, source and destination ports, protocols, and bytes transferred. Specifically, we need a NetFlow analysis tool that is capable of showing us connections between our internal systems and systems on the Internet. In our case, we use a product called LYNXeon to do this. Alone, LYNXeon does a good job of allowing us to visualize connections from one system to another, but finding the systems related to malware issues can often be a needle in a haystack because of the NetFlow limitations mentioned above. So while our malware connections (downloads and command-and-control) are buried in the NetFlow data, we really have no way to identify them in the NetFlow tool silo.
Now comes the fun part. One of the cool things about the FireEye system is that it provides us with the ability to export data and one of the cool things about the LYNXeon system is that it provides us with the ability to import data and tag it. So what we do is, in FireEye, we export the list of all systems that we have detected as having been infected by malware. We also export the list of all of the command and control servers and malware hosting servers that we have seen connections to. Next, we go into LYNXeon and tell it to import these two lists of IP addresses and tag them with a custom tag that we created called "FireEye". We have now successfully combined these two tools and the payoff is huge.
Success #1: Detecting the Spread of Malware on Your Network
Our FireEye system works by executing downloads inside of a virtual machine and analyzing the affect they have on the system. Because the virtual machine doesn't always match the target system, in many cases we are only able to tell that it was malware and not that the malware actually infected the system. Using LYNXeon, however, we can create special queries that will show us all connectivity from the potentially infected system after the time of the malware download. Did the system immediately make connections to other foreign systems on the Internet? Did it start scanning our internal network looking for other hosts to compromise? All this and more is possible now that we have identified a potentially infected system on our network. Here is a pattern file which I created in LYNXeon to do this:
And here is the pattern diagram which this query accomplishes:
Success #2: Finding Other Infected Systems
FireEye appliances aren't free and with offices in over 40 countries around the world getting full coverage can get expensive. But, if we can use a handful of appliances to get an idea of where our systems are talking to when compromised, then we have data which we can turn around and use in places where we do not have those appliances. Because we are sending NetFlow data from our devices around the world into LYNXeon, we can search for any connections to these common malware servers. No more needle in a haystack. The data is all there, we just needed to know how to look for it. Here is a pattern file which I created in LYNXeon to do this:
And here is the pattern diagram which this query accomplishes:
Success #3: Discovering Other Types of Attacks
Often times our adversaries aren't just trying one type of attack and giving up when it fails. They are trying every trick in their arsenal and trying to gain and maintain a foothold on your network with whatever method they can. Once we've identified an attacker's IP address, we can now use our NetFlow data to see all other traffic coming from that IP address. Often times, expanding these types of relationships can shed light on other activities they are performing on your network. Perhaps they are performing reconnaissance on your servers? Maybe they are trying to DOS one of your systems? The fact is that once they've been uncovered as a bad guy on your network, you should be weary of all activities performed by them. Maybe even ban their IP address altogether. Here is a pattern file which I created in LYNXeon to do this:
And here is the pattern diagram which this query accomplishes:
So there you have it. By combining our malware analysis using FireEye and our NetFlow analysis using LYNXeon, we have created a hybrid system capable of far more than either of these tools by themselves. This is the magic of symbiotic security in action. Our tools becomes infinitely more powerful when we are able to share the data between them. Hopefully you will take that into consideration the next time you are looking at purchasing a security tool.
About a week ago I turned on a new rule on our IPS system that is designed to detect (and block) users who are using TOR to make their activities on our network anonymous. You can say that TOR is about protecting a user's privacy all you want, but I'd argue that while using corporate assets you should have no expectation of privacy (at least in that sense) and that the use of anonymizers on a corporate network can typically be viewed as a sign that you are up to no good. Almost immediately when I turned on this new rule, I began seeing associated events in the IPS console. I decided that the best approach was to contact the user directly as they may be wondering why their Internet connection was no longer working. I reached out to this particular user and explained that if this was the case, then it was because of the new IPS rule. The solution was simple; just reconfigure his browser to no longer use TOR as the proxy. But as I began this process, things started getting weird.
I began by telling the user to look for names like "TOR", "The Onion Router", and "Privoxy" in his Add & Remove Programs. Strange....there was nothing there. Then I asked him to check his Task Manager to look for a running process called "tor.exe" or similar. Again, nothing. I was at a loss. I decided that this was something I needed to get my hands on to figure out so I scheduled some time with the user.
This morning when I sat with the user, I noticed little wrong with his system. He had a few standard applications running, but nothing unusual. I checked his process listing and saw nothing out of the ordinary. I ran Hijack This! and that, too, looked pretty normal. All this, yet in the meantime I continued to see alerts on the IPS system that his computer was using TOR. Even when I was sitting at the console with NO browser activity. So, to make a long story short, here's how I finally figured out what was happening. I checked the IPS system and came up with the source ports for the requests that I was seeing alerts on. I then went on the system and ran a netstat -nao. This listed all network connections on the users system along with the associated process. I checked the list and found the entry that matched the port number I was seeing the alerts on. I then ran the command tasklist /svc /FI "PID eq <process_num>" This provided me with the name of the process that was running with this process ID which it turns out was "iexplore.exe". Wait. Internet explorer isn't even running on this computer. Or is it? Since the default process viewer in the Task Manager is pretty lame, I downloaded the Microsoft Sysinternals Process Monitor. It's a free tool available from Microsoft and provides a ton more information about running processes and allows you to see what they are doing in real time. I used the Process Monitor to view these processes and focused particularly on the flags that were used when they started. What I found was actually pretty startling.
Both of the Internet Explorer processes were started with a special flag that told them to start silently (ie. without the UI) in the background. They also specified a flag similar to this:
--HiddenServiceDir "C:\Documents and Settings\<User_Name>\Application Data\tor\hidden_service" -- HiddenServicePort "55080 127.0.0.1:55080"
Aha! We found our culprit! TOR was running as a hidden service out of the Application Data directory. Once I found this, it was all over. Scanning through the Application Data directory, I also found a file under "Enemvy\ugbie.exe" that was extremely suspect. A later scan via Malwarebytes identified it as a variant of Trojan.ZbotR. I deleted these directories and Malwarebytes found one registry key associated with the ugbie.exe file and deleted it. All is good now and the system is no longer alerting about use of TOR.
So, what's our lesson here? The malware writers are getting sneaky. They've realized that we've created blacklists of their servers and they need to be able to adapt around that. Now, they are using anonymizers, like TOR, to get around these blacklists. Apparently this isn't the first use of TOR in malware either as I read about something called SkyNet that did something similar. In any case, they would have gotten away with it if it weren't for my IPS rule to detect TOR and a fair amount of persistence in finding the root cause. If you're not already detecting this on your network, I think that it's about high time you did it. You can thank me later.
This post is going to be short and sweet as it's something I meant to put up here when I found it sometime back in mid-2011. I'm not even sure if Time Warner is still using these Ubee cable modems for their RoadRunner offering, but I'm sure that there are at least a few people out there who still have them. When you get the modem installed initially, they give you some default credentials. Something like user/user or admin/admin. Using these credentials, you are able to access the device and many of the features that it has to offer you. What you are not able to do is access the menus where you can change how the router is actually configured for internet access, change the master password, or prevent Time Warner from accessing your modem, and subsequently, your network. To fix this, you just need to know the following secret...
The real administrator username that comes configured on these modems when you get them from Time Warner is the last eight digits of the unit's MAC address sans the colons separating out the values. This is unique to your device, but can be found pretty easily by looking at the user interface that you do have access to. The password for this user is "c0nf1gur3m3". Use that and you should be in. Feel free to change the password while you're in there to keep the Time Warner folks out.
One other kinda secret thing to note is that if you do want to change how the router is configured for internet access, you will need to go to http://192.168.0.1/TlModeChange.asp on your router to do so. Once there, you can change it to Bridge mode, NAT mode, Router mode, or NAT Router mode depending on what you are looking to do with it. Hope you enjoyed this simple solution for getting the real administrator access to Time Warner RoadRunner's Ubee cable modem.
***Update: If the above isn't working for you on Time Warner Cable, try one of these suggestions from the comments:
- Username: admin / Password: cableroot
- Username: technician / Password: C0nf1gur3Ubee#
- Username: admin / Password: C0nf1gur3Ubee#
A couple of weeks back, HD Moore posted a blog entry entitled
"Security Flaws in Universal Plug and Play: Unplug, Don't Play" supporting a Rapid7 Whitepaper in which he discusses the 81 million unique IP addresses that respond to UPnP discovery requests on the Internet and the 23 million fingerprints that match a version of libupnp that exposes the systems to remote code execution. His research on the subject is fascinating and I highly recommend reading it over, but that's not the reason why I'm writing this. The first question this research had me asking myself is whether or not my organization utilizes UPnP for anything. As far as I can tell, the answer to this question is, thankfully, no. Next, out of curiosity I began to wonder how many people were out there actively trying to find these exploits. A perfect opportunity to fire up our new LYNXeon tool.
Our LYNXeon tool is configured to consume NetFlow data provided by literally hundreds of routers and switches in our global environment. One of the most interesting things about it is that it can be used to see the traffic that comes in from our edge routers before it gets squashed by our firewall. Utilizing this tool in this way, we can visualize the so-called "Barbarians" at our gates. These are the hackers that are out there trying to find the weak spots in our security in order to get in. And since I know that UPnP is not a service that we offer up to the Internet at large, it makes finding the guys who are looking to exploit it that much easier.
I fire up LYNXeon and my first step is to generate what is known as "PQL" or "Pattern Query Language". While their Cyber Analytics Catalog offers up a ton of templates to use to find potential threats, PQL is the base of all those queries and writing your own allows you to define your own catalog of things to look for. The language is pretty easy to understand. First you define the characteristics of the connections that you are looking to find. After doing some research, I found out that HD was looking for openings on UPnP's Simple Service Discovery Protocol (SSDP) service which typically runs on UDP/1900. So, my query is for connections from external source IPs to internal source IPs using the UDP protocol on port 1900. Once the connections have been defined, all that is left to do is define the data that you want to see in the results. In total, my PQL code is 15 lines of code:
Now it's officially time to make these invisible Barbarians visible. I tell LYNXeon to only show me results over the last day (to reduce the amount of time the search takes) and then tell it to "Execute Pattern Search" using the pattern file that I just created. Searches will vary in time based upon the timeframe searched, the number of forwarding devices, and how complicated your search criteria are. For me, this search returned 539 results in one minute and 38 seconds.
Now that I have results, I just need to select how to view them. My personal favorite is viewing the results in the Link Explorer. This will show my data as nodes on a pictoral graph. I make one quick adjustment using a organizational feature called "Force Directed Layout" to make the pictures look pretty and voila!
OK, so zoomed out it looks like a bunch of spider webs. Now the fun begins as we begin zooming in on each cluster to see what is going on.
I've blacked out the IP address of the system these guys are connecting to as it is irrelevant for the purposes of this post, but you can clearly see that in the past day this one system has had eight unique IP addresses attempt to connect to it on UDP port 1900. I've got dozens more just like these on that big graph above with varying degrees of complexity. From here, LYNXeon allows me to resolve DNS and/or ARIN names for the associated IP addresses. I can also expand upon those sources to see what else of mine they've been talking to. Is that cool or what? It's taken me minutes to find these potential threats and with little more than a few clicks of the mouse. The Barbarians are most definitely at my gates silently pounding away and chances are pretty good that they are doing the same to you. The question is....can you find them?
I was having lunch with Charles Henderson from Trustwave Spider Labs the other day and he mentioned that he had just gotten signed up with the new Roadrunner Extreme Broadband Beta from Time Warner Cable. He mentioned insane download and upload speeds as well as the new DOCSIS 3.0 compliant modem. It was enough to pique my interest and get me to call Time Warner.
I have been on the older Roadrunner Turbo-charged plan since basically when it first came out and have been generally happy with the service up until recently when I've started having to reboot the modem daily. I'm also kind of an internet speed addict so the idea of moving up to 20 MB/s downloads and 5 MB/s uploads was pretty sweet to me. That's just to start with as eventually the service will have 30 MB/s downloads. I called up Time Warner and asked what it would take to move onto the Extreme Broadband Beta and they told me that it was only an extra $5/mo over my Turbo-charged plan. Even better was that they were offering free installation as part of the Beta. They were able to get the install scheduled just over a week out. Not too bad.
The service technicians came out on the designated day and time and got everything hooked up for me. They even replaced a bunch of the wiring on the box on the side of the house where the service connects to. They did some line tests and within minutes I was up and running on the new service. While not the 5 MB/s upload that was advertised to me, the download speed is quite impressive. Check it out:
The other cool thing is that while not necessarily intended, it is very easy to get into the new ubee modem's configuration interface. By default, the device comes up as 192.168.0.1 on your network and has a username and password of user/user. Get in there and it's got all of the configuration options of a wireless internet gateway. The first thing that you should do is change the username and password. After that, enable the wireless network, configure port forwarding, etc.
Not only does the new modem have built-in wireless N, but it also has four additional network ports so you can use it with multiple computers on your network. I remember the days when Time Warner used to charge you if you had more than one computer, but not anymore.
Granted, I've only had the new service for a few hours now, but I'm already pretty impressed. If you're an internet speed demon like me, and you live in the Austin area, I'd recommend that you give Time Warner a call and ask about switching over to the new Roadrunner Extreme Broadband Beta. Enjoy!
I was talking with my coworkers this afternoon about Time Warner's plans to jack up rates for high-bandwith users and it got me thinking about how much of their precious bandwith I am actually using. I know that my router at home has a web browser interface where I can get that information, but I have it intentionally only allowing access from the local area network interfaces. I needed to find another way to view the site from work while making the router think that I was on the right network. What I ended up doing was using PuTTY to create a SSH tunnel from my work computer to my Linux box on the home network. I then just pointed my browser at the forwarded port on my work computer and up comes my router's web interface. Who needs VPN when you have PuTTY? Anyway, here are the exact steps that I took to do this:
- Start PuTTY
- Under Connection->SSH->Tunnels specify a source port (the localhost port you want to connect to) and a destination (IP:port) that you want to connect to on your home network.
- Source port: 8008
- Destination: 192.168.0.1:80 (or whatever IP your router is at and it's web interface port)
- Click "Add"
- Under "Session" specify the host name for your SSH server that lives on your internal network, but is exposed via port forwarding on your router with port 22.
- Click "Open"
- When prompted, enter your username and password for your SSH server.
- Now just pull up your favorite web browser and navigate to http://localhost:8008. You should see the page just like you would if you were sitting at home.