Web Admin Blog

Real Web Admins. Real World Experience.

Why I’ve Lost All Faith in Verkada

About two years a sales rep from a company called Verkada reached out to me to evaluate their approach to physical security monitoring for our campus. While our physical security isn’t under my team’s purview, it’s something I’ve always been interested in and we’ve helped to advise the facilities team responsible for it in the past. I decided to take them up on their offer to demo one of their cameras.

The device itself looked pretty cool. It appeared to be a high quality camera with capabilities to zoom in and focus and it came with hardware to mount it outside. The place I identified as an ideal location to mount it was high up above my driveway on the front side of the house and I hired someone to come out and run the wiring for it. A single ethernet cable with power running through it as well.

Once I got it up and running, I was very impressed. It was a decent quality camera that was constantly recording video to the cloud and allowed me to easily hone in on events of interest. Things like when my brother-in-law invited a girl over to our house while we were away on vacation or the kids who stole our bowl of Halloween candy. For over a year and a half it did exactly as advertised and I sang it’s praises to many people because of that. Then, this past April, the camera stopped recording and all I saw was a message saying it was updating.

I waited for days…weeks…over a month for the update to complete. I tried rebooting it several times as the Verkada website talks about how they keep a backup copy of the firmware so the device is virtually unbrickable. Nothing worked so eventually I reached out to Verkada support.

After some light troubleshooting around the color of the lights on the camera and what they were doing, support blamed it on my firewall. They said that it was “internet access related”. Must be the firewall blocking access to their cloud endpoints, NTP, or DNS resolution. The finger pointing had begun.

If you’ve read any of my blog posts, you probably know that I’m no dummy. I had already been reviewing my firewall configuration and logs and had determined that the DNS traffic was going through just fine and the only oddity was that the client side of the connection it was making to their servers showed an odd “client-rst” action. They asked for a packet capture, which I sent to them, and concluded “the firewall or something upstream is doing some SSL inspection which is affecting the connection”. I had created an isolated policy to help troubleshoot the issue that had no rules other than “allow all ports and protocols from this device to any location”. They claimed that “on the backend we are not seeing successful connections from the camera in regards to data contained in the TLS tunnel” (something that turned out to be outright made up because I showed responses from their servers in the packets). They said “this looks like the firewall is blocking the camera from making a TLS connection to the cloud”.

All of my troubleshooting at this point pointed to issues with the camera closing the connection, but they continued to point the finger at my firewall so I submitted a support ticket with Fortinet support. Fortinet support responds with “Looking at the output there are no drops on this capture. The firewall has distinct messages that it will provide when it drops a packet, indicating why. The example traffic you sent me shows the 3 way handshake completing before the Reset Packet originates from [the device]… Based on the debug flow, the Firewall is not dropping the traffic, and the traffic based on the successful complete of the 3 way handshake is flowing in both directions and received by all the involved devices… After the completion of 3 way handshake we are seeing the Reset coming in from the Client device. The Packet comes in as a Reset from the Client, on port2. Note that if the Fortigate was intervening and sending the reset on the Clients behalf it would show it originating from “Local” instead of Port2.”

At this point, I figured out how to turn off ASIC offloading on the firewall and got a full PCAP of what was going on. I noted a ton of “Certificate Expired” messages each time the client tried to negotiate a TLS connection with the server. Several more PCAPs later and the Verkada support person eventually tells me “I believe the camera may be experiencing a bug with system that affected a small population of cameras in the wild. We have since patched this issue but on rare occasions a camera cannot be recovered.” At this point I’ve spent several days troubleshooting an issue that they knew about, but instead pointed fingers at my firewall vendor. I’m kinda upset about how we got here, but at least we’re at the point where they’re going to RMA it and send me a device that works properly, right? Nope.

The following day I get a final email from the Verkada support representative telling me “I submitted an RMA but it was denied. The reason being, the camera was a beta giveaway and was never paid for.” I get it, I didn’t pay for it, and I can respect their desire to not want to pay to ship me another one. That being said, I accepted this under the pretense that Verkada was providing me with a unit to evaluate and provide feedback to my facilities team and others in the InfoSec community. I provided positive feedback when it was up and running, but now that it’s gone belly up I feel like I need to share that as a data point for perspective customers, now, as well.

I guess that one could argue that a customer would have paid for the device and the eventual RMA would have resulted in a new one being sent, but these aren’t being sold to techies like me. These are being sold to businesses who want an easy way to monitor what is going on around them. Businesses who don’t have the time or ability to send PCAP after PCAP, the expertise to push back against the finger pointing by support, and the ability to involve support from their firewall vendor to get to that resolution. And even if the business got to the RMA, do they really want to have to go through the hassle of pulling all of the devices down and putting new ones up?

At this point, I’ve lost all faith in Verkada’s product and their ability to properly support enterprise customers. I rescind all recommendations that I’ve made for them in the past and would highly encourage others to look elsewhere for their IP camera purchases.

Completing the LASCON 2018 Badge Game

The LASCON Badge Game was back in 2018 and the feedback I received was that it was the best one so far. It started out with the following QR code on the back of the badge:

Following that QR code took you to the URL https://pastebin.com/J61kDSe2. Viewing that URL gives you the following:

V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSG=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSE=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSH=
V2VsY29tZSB0byB0aGUgTEFTQ09OIDIwMTggYmFkZ2UgZ2FtZSF=

These are clearly BASE64 encoded strings and if you do a BASE64 decode on any of them, you get text that reads “Welcome to the LASCON 2018 badge game!”.

Each of the BASE64 encoded strings represents a 00, 01, 10, or 11 value, which is represented by changing the last character of the BASE64 encoded string. To learn more about this form of steganography, take a look at this CTF writeup.

Translating each of the strings in pastebin into a 00, 01, 10, or 11 value will give you a binary string value of:


Converting that string to ASCII text, you get:

NTQuMjI2Ljg5LjEyMA==

This looks like another BASE64 encoded string and decoding it gives you 54.226.89.120, which is obviously an IP address.

If you’re following along with me this far, you’ve gotten as far as you can go as the server hosting the badge game is no longer online. However, when you went to 54.226.89.120, you saw a file named lasconbadgegame.pcapng.zip. This was a packet capture file with a bunch of random web surfing contained in it, but if you isolated the requests to the 54.226.89.120 IP address, you would see a request for port 7777, port 8080, port 9797, and finally port 31337. The system was running a port knocking application that would open a hole in the firewall to port 31337 for any IP that hit the other three ports first, in the proper order.

Under http://54.226.89.120:31337 there was a submit.php page with a login prompt, a winners.php page that shows a list of who has won the game so far and when, an encrypted submission_password.zip file, and a rockyou-20.txt file which contained the RockYou word list.

You needed to take the rockyou-20.txt file and brute force it against the submission_password.zip file. Here’s what it looked like when I ran it using the fcrackzip application:

Using the found password of “147258369” to decompress the submission_password.zip file, the resulting submission_password.txt file contained the following text:

Looking at the submit.php page source code, there’s a comment line that reads:

Simply log in to http://54.226.89.120:31337/submit.php with the username “lascon” and password “P6KKWZQeKvMvTvBGNsvX” and submit your personal information to win. That was it! Another awesome LASCON Badge Game in the books and most I spoke with said it was the best one yet. Winners received this awesome LASCON 2019 Badge Game Challenge Coin…

Completing the LASCON 2017 Badge Game

For those who don’t know, every year I put together a game that starts on the back of the LASCON badge.  It’s typically some combination of crypto challenges alongside application security vulnerabilities with the goal of having it take somewhere around 1-3 hours, depending on experience, to complete.  Those who complete the game are rewarded with one of these awesome challenge coins:

LASCON 2017 Challenge Coin

Now, I know that there are people out there who look at one of these things and don’t even know where to start so, it is in the spirit of education and learning that I share with you my notes on how to complete the LASCON 2017 Badge Game.

Stage 1

On the back of the LASCON badge it reads as follows:

Another year, another game
Solve the puzzles, write your name
These characters aren’t a work of art
Ask a mason how to start
ciphertext
The key word in the text above is “mason”.  If you were to Google the term “mason cipher”, you would come across an interesting kind of cipher called a Pigpen/Masonic/Freemason Cipher.  The idea being that they take a geometric pattern and map the letters of the alphabet to the locations on the pattern.  Here’s an example that you could use to translate this text:
Freemason Cipher
Once translated, you get the following message:
To start the badge game go to nocsal dot lascon dot org
Stage 2
When you go to nocsal.lascon.org it defaults to having a GET parameter of page=winners.php.  This is a sign that it is vulnerable to directory traversal.  There is also a comment in the page source that says “Get badge game winners from /files/winners.php”.  If you navigate to http://nocsal.lascon.org/files/, it has directory browsing enabled and you can see a test.php page there, in addition to winners.php.  If you go to http://nocsal.lascon.org/?page=test.php, you will see it grabs the test.php code and pulls it into the page source.  If you view source, you can see the text is as follows:
<?php
// Test ability to grab winners table from the database
$servername = “localhost”;
$username = “lascon”;
$password = “e3fmGYHDrc6MNCEMmLWj”;
$dbname = “lascon”;
$conn = new mysqli($servername, $username, $password, $dbname);
$sql = “SELECT * FROM lascon_winners”;
$result = $conn->query($sql);
$array = $result->fetch_array();
print_r($array);
$conn->close();
?>
We now have a database username and password.

Stage 3

Even though the database connection that we found uses a servername of “localhost”, it turns out that mysql is open on the public IP interface of the server as well.  We can use a mysql client to connect to nocsal.lascon.org with username “lascon” and password “e3fmGYHDrc6MNCEMmLWj” (mysql -h nocsal.lascon.org -u lascon -p).  Once in the database, the user has access to read the lascon database and will see a “lascon_winners”, “users”, and “websites” table.  When you try to insert into the lascon_winners table, you quickly realize that you do not have insert permissions, so you cannot insert into the lascon_winners table.  In the users table, you see an entry with username “admin” and password “NFJXaTNuc0pyR2Y2c25iNG9Va1c=“.  In the websites table, you see a bunch of sites and one hiding amongst the others is ttpcteebhz.lascon.org.

Stage 4

When you go to http://ttpcteebhz.lascon.org in your browser, you see a form with a spot for a username and password.  You can base64 decode the string you found int he database (NFJXaTNuc0pyR2Y2c25iNG9Va1c=) to get the value “4RWi3nsJrGf6snb4oUkW”.  Once you have that, you can log in with username “admin” and password “4RWi3nsJrGf6snb4oUkW”.

Stage 5

Once logged in with the username and password, you see a blank page.  Once you view the page source, however, you see that it contains a hidden form and fields:

<form name=”submission” method=”post” action=””>
<input type=”hidden” name=”first_name” value=”” readonly />
<input type=”hidden” name=”last_name” value=”” readonly />
<input type=”hidden” name=”phone” value=”” readonly />
<input type=”hidden” name=”email” value=”” readonly />
</form>

The last part of the challenge you could accomplish with a proxy tool, but I just used the Developer Tools in Chrome.  I changed the hidden fields to text fields, removed the readonly values, and then added a form submit button. Once submitted, the game is over and you win!

A Quick Summary of Puzzles Solved / Vulnerabilities in the Badge Game

  • Freemason Cipher
  • Directory Traversal
  • Information Disclosure in Comments
  • Directory Browsing Enabled
  • Hard-Coded Database Credentials
  • MySQL Service Publicly Accessible
  • BASE64 Encoded Passwords
  • Hidden and Read-Only Form Fields
  • Missing Form Submit Button

Completing the BSides Austin 2016 Mini-CTF

The BSides Austin 2016 Mini-CTF began with the back of the badge.  There was a large QR code which took a very long time for me to scan with my phone, and when I finally got it, it was just the numbers “07263584”.  Not very useful.  Below that, however, there was  a string of letters and numbers as follows:

aHR0cDovL2N0Zi5ic2lkZXNhdXN0aW4ub3JnL2xldmVsMS8=

The string very obviously looked like BASE64 so I decoded it to get the following URL:

http://ctf.bsidesaustin.org/level1/

On that page it reads:

That start was easy! You have shown that you are curious and that is the key. As your reward, you may have this:flag 1: BSides{D3c0d3s_R_3Z}Do you want to play some more? If so, read on…

1. Turn in flags by sending an email to bsidesaustin@gmail.com. The email must contain your name, email address, and the flag you are turning in

2. There are three flags total, each should be submitted via email.

3. Do not scan this server with automated tools. They are not necessary and could cause performance issues. If you scan this server, you could be disqualified.

4. Send in flag 1 then click here to continue…

I submitted the flag and moved on to the next page at:

http://ctf.bsidesaustin.org/level2/9slfowiuwer98987987kljsdfljsdf/

On that page there was a file named coms.pcap.  With the “pcap” extension, I went ahead and loaded up into Wireshark.  It was a 1113 line packet capture file with encrypted Google traffic, YouTube rick rolls, and more.  Only a handful of the requests were for 45.32.195.232, the IP address belonging to ctf.bsidesaustin.org.  When I followed the TCP stream, it was for a request to http://ctf.bsidesaustin.org:31337/level3/index.html.  When I went to that URL, however, it was as if nothing was listening.  Eventually, I filtered the pcap by that IP as the destination and found a sequence of requests to that IP at odd ports…1025…2300…1337…1337.  This smelled suspiciously of port knocking so I wrote a quick bash script using nmap to test it out:

for x in 1025 2300 1337; do nmap -Pn –host_timeout 201 –max-retries 0 -p $x 45.32.195.232; done

Finally, I hit the page on port 31337:

http://ctf.bsidesaustin.org:31337/level3/index.html

Sure enough, now I get a response that says:

Congratulations, you have completed the second challenge!The second flag is: BSidesAustin{C4rV1NG_UP_PC4Ps}

Click here to continue to the final challenge!

I submitted the flag and moved on to the next page at:

http://ctf.bsidesaustin.org:31337/level3/owiroewuouoiu

On this page we see a network interface test where you can specify an IP as a ping destination, and when you hit “Enter”, it gives you the results of the ping:

PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.026 ms— 127.0.0.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms

rtt min/avg/max/mdev = 0.026/0.026/0.026/0.000 ms

Entering some non-IP address information, I got a message like:

Error running ping -c 1 127.0.0.1

So, there is some filtering on it, but it also looks like some data is making it through.  I figured out that I could specify the destination as a GET instead of the POST with by adding “?dest=127.0.0.1” to the URL and that worked.  I tried a bunch of different combinations of “;”, “&&”, and other OS command functions that would piggyback on the existing function with no luck.  Eventually, I figured out that “%0A”, the ASCII line feed control character, was not filtered and I could use that to run more commands.  For example:

http://ctf.bsidesaustin.org:31337/level3/owiroewuouoiu/index.cgi?dest=127.0.0.1%0ls

Returned a listing of “index.cgi” and “the” in that directory.  Then:

http://ctf.bsidesaustin.org:31337/level3/owiroewuouoiu/index.cgi?dest=127.0.0.1%0find

Showed that “the” actually expanded to “/the/roof/the/roof/the/roof/is/on/fire/flag.txt”.  Sweet!  I found the flag!  Now to open it.

http://ctf.bsidesaustin.org:31337/level3/owiroewuouoiu/index.cgi?dest=127.0.0.1%0Acat%3C./the/roof/the/roof/the/roof/is/on/fire/flag.txt

That returned:

PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.026 ms— 127.0.0.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.026/0.026/0.026/0.000 ms
Great job! The third and final flag is:BSidesAustin{F1lt3rs_R_Fun}

Congratulations, you have completed the challenge!

Final flag submitted.  Game over!

What Powerball, Poker, and SimpleRisk Have in Common

With an estimated Powerball Jackpot of $1.5B, everybody is talking about it right now.  I’ve got my tickets, but with a prize that big, I’m having an easier time listing the things that I can’t buy with the money, rather than what I can.  That said, I keep coming back to a concept that statisticians refer to as “expected return”.  What it means is that even though your odds of winning the Powerball aren’t changing (roughly 1 in 292 million), the bigger the jackpot, the bigger the expected return.  To calculate it, just take the odds and multiply by the reward value.

1/292m x $1.5B = $5.14

So, before you factor in taxes, the possibility of splits, and other factors that might affect your reward, the expected return is something like $5.14 on a $2 ticket.  Not too bad.  Factoring in the cash option ($930M), a 40% Federal tax, an extra 25-28% tax from the IRS for gambling winnings, and perhaps even state taxes, however, this number drops well below the $2 range making it far less appealing to buy a ticket.  I found interesting articles in Business Insider and Wired that elaborate further on this idea.

The same concept can be applied to poker in what they refer to as “pot odds”.  You take the odds of a winning hand and multiply it by the size of the pot.  In poker, often times players are more willing to play what is normally a statistically losing hand if it gives them a shot at more money (ie. a bigger pot).

So what does this have to do with SimpleRisk and risk management?  The classic formula for calculating risk is RISK = LIKELIHOOD x IMPACT and is the exact same formula used for calculating expected return and pot odds.  You’re simply taking the likelihood of an event happening regardless of whether it is a positive or negative outcome and multiply it by the estimated dollar value of that outcome.  Pretty simple.  And that is what Powerball, Poker, and SimpleRisk have in common.

The OWASP Board “Ivory Tower” Dilemma

I have been an active member of the OWASP community in some form since 2007.  I’ve been the OWASP Austin Chapter Leader, served as the Chair of the Global Chapters Committee, and, most recently, was elected (and re-elected) to the OWASP Board of Directors.  In the past, I have heard a number of people in our community compare the Board to an “Ivory Tower”.  They would say that they were unapproachable and preferred to let others do the work while they pulled the strings of the Foundation from behind the scenes.  I think there may be some truth to that statement, but I told myself when I ran for the Board that I wouldn’t be like that.  I want people to feel like I am out there actively trying to solve the problems of our community.  Case in point, in my 23 months on the OWASP Board, I have proposed more Bylaw changes and new policies than anybody else.

As I continue to try to be a man of action, I find that I am often one of the first Board members on the scene in times of crisis.  On multiple occasions I’ve shunned the historical “Ivory Tower” approach to managing the organization and dove head in to the situation at hand.  My assumption has been that I was elected because I have an opinion, not in spite of it.  In each case I’ve tried to present a clear and concise analysis of my view of the situation.  I’ve tried to offer up suggestions on next steps or provide data points that others may not have been aware of.  Being such a diverse community has many strengths, however, one weakness is that it is difficult to drive to a consensus on anything.  It really doesn’t matter what side of the issue you are on, it always seems like some will agree with you and others will not.  Frequently, what begins as an intended friendly and spirited debate, ends with somebody feeling marginalized because a decision was made that they did not agree with.  It’s sad when this happens, but is inevitable when you mix passionate people with issues that do not have binary answers.

This leads me back to the “Ivory Tower” dilemma.  If my desire is to actively be a part of the community, then I place myself directly in a position of potential conflict when I speak.  I’m not allowed to speak as Josh, the community member, because the perception is that I am always speaking with my Board member hat on.  And I have a strong feeling that this perception of Board members speaking authoritatively is what leads a person on the other side to feel marginalized.  Definitely not intended, at least on my part, but that’s what I’ve started to gather from some of the feedback that I’ve received.  So if that’s the case, then I begin to wonder if the situation would have been better off had I held my tongue and refrained from jumping into the discussion in order to let our community continue to fight it out or to let another Board member, our Executive Director, or somebody else communicate the Board’s analysis and actions.  But, if I do that, aren’t I now perpetuating the stereotype of the OWASP Board being an “Ivory Tower”?

I’m not sure that there is a right or wrong answer here and nobody said that being a Board member would be easy, but I can’t say that I ever expected to need to give up my personal voice with the community (the one that likely got me elected to the Board in the first place) in order to serve the Board.  That said, it genuinely saddens me when an extremely valued OWASP volunteer feels the need to leave in order to make a point.  It is a huge loss for the OWASP Foundation and one that I, regrettably, played a role in provoking.  I don’t apologize for my stance on the issue that was being debated.  I feel that we should all be allowed to have an opinion and I still support the actions of the Board thus far.  That said, if I could take back my words, crawl back into that “Ivory Tower”, and let someone else do the talking in this particular situation, I’m sorry to say that I would.

Johanna, I’m sorry that it turned out this way.  You may not believe it, but I sincerely respect and appreciate what you have done for the OWASP Foundation more than words can express.  You have brought order where there was chaos and a dedication to the cause that was matched only by your intellect.  I feel that we don’t have to always agree on a vision in order for me to appreciate your perspective.  I regret that I never conveyed that to you before now.  I’m sorry.

SSH on a Mac Errors with “Write Failed: Broken Pipe”

Recently I had an issue when moving to a new Mac on OSX when I was trying to SSH to a Linux server.  It would make the initial connection and then prompt me for a password.  Once I entered the password, however, it would just hang (ctrl+c wouldn’t even escape out) until eventually it would break to the command line with the message “Write Failed: Broken Pipe”.  After digging through various posts online, most of which were referring to timing out due to inactivity, I finally found a winner.  I edited the /etc/sshd_config file and set the ClientAliveInterval value to 300.  Then, I rebooted my Mac.  The next time I tried to SSH to the same server, everything connected as expected.  I hope this helps someone else in the future who is running up against the same issue I had.

Fixing when crashplan won’t start anymore on your Drobo

Even though the Drobo is supposed to be a pretty rock-solid tool for backing up your files, there are still plenty of reasons why one would want to keep a copy of those files elsewhere just in case.  For example, what would happen if there is a fire and your Drobo is damaged.  Are you OK with losing everything?  I’ve even heard about the rare case where the Drobo drives get out of sync and a complete reformat is necessary; causing you to lose everything.  To prevent this, it is a good idea to install the Crashplan Drobo app and ensure that a copy of your data is recoverable, even if the worst case scenario happens with your Drobo.

If you do as I mention above, chances are that things will work well for a while and then suddenly, one day, you will find that Crashplan is no longer running on your Drobo.  Despite multiple attempts to start it back up, you will inevitably find yourself staring at a message saying either “crashplan is enabled and stopped” or “crashplan is disabled and stopped” and will be clueless, like I was, in terms of how to fix it.  The good news is that after months of struggling with this, I finally came across a post on the DroboSpace forums from the guy who packages the Crashplan application for Drobo.  It was a bit cryptic at first, but eventually I was able to interpret what he was saying and I wanted to share it with everyone in a bit more layman’s terms.

The underlying issue here is that Crashplan is configured to automatically upgrade itself on the Drobo.  When this happens, it downloads the replacement files and goes to run the upgrade script.  Unfortunately, the Crashplan team does not write the upgrade script to work in the Busybox environment (the one that runs on your Drobo) and the script breaks.  By tweaking the script ever so slightly, you can get it to run the upgrade and Crashplan will once again start up on your Drobo.  Here are the steps to do it:

  1. SSH into your Drobo with the command “ssh -L 4201:localhost:4243 root@[your Drobo IP]
  2. Take a look at the /tmp/DroboApps/crashplan/log.txt file and you’ll probably see a message saying something like “Error: Could not find or load main class com.backup42.service.CPService”
  3. Go to the crashplan upgrade directory with the command “cd /mnt/DroboFS/Shares/DroboApps/crashplan/app/upgrade
  4. Here you will see a directory whose name looks like a random value like “1388642400364.1415587076262”.  I believe you should see a new one of these directories for each version you are upgrading to.  Change to that directory using the command “cd 1388642400364.1415587076262” substituting for whatever directory name you see.
  5. Edit the upgrade.sh script inside that directory.  You want to change the “rm -fv” line to “rm -f” and the “mv -fv” line to “mv -f”.  You will also want to search for the two lines that start with “/bin/ps -ef” and change them to use “/bin/ps w” instead.  Save the file.
  6. Change the permissions on the upgrade.sh script to make it executable with the command “chmod +x upgrade.sh“.
  7. Run the upgrade.sh script with the command “./upgrade.sh“.

When the script completes, you should be back at the terminal prompt.  From here, you can go back to the /mnt/DroboFS/Shares/DroboApps/crashplan directory and try starting Crashplan using the command “./service.sh start“.  Check that it is “enabled and running” by running the command “./service.sh status” to get the status.  You may have to run through steps 4-7 multiple times based on how many upgrades back you are, but when all is said and done, you should be back up and running with Crashplan on your Drobo.  Good luck!

Why You Shouldn’t Phish Your Users

As an Information Security Program Owner, I get a barrage of e-mails and phone calls multiple times a day from vendors looking to sell us their latest hotness security product.  Between the e-mails, phone calls, expo floor at BlackHat this year, and several talks that I’ve seen at past conferences, I have noticed a disturbing trend that I thought was worth bringing up: phishing your users.

The concept is simple; you send e-mails to your users with content that appears legitimate along with links or attachments that are designed to simulate a spear-phishing attack.  If the user recognizes it as malicious and deletes it, then they are left to carry on with business as usual.  If, however, they fall victim to your trickery, then they are punished in the form of verbal and written lectures, letters to their management, and security awareness training.  No carrot, all stick.  This situation makes me think back upon an issue that I’ve encountered with my twin daughters at bedtime.  For over a year we struggled to get them to stay in bed at night.  We would lay them down, play some music, and then leave the room and it wasn’t 5 minutes later before they were up playing, yelling, and coming back out into the hall.  We yelled at them, spanked them, turned off lights, and did just about everything we could think of to get them to stay in bed.  None of the punishments we did actually corrected the behavior.  Do you want to know what actually worked?  Offering them a treat in the morning if they stayed in bed all night.  Positive reinforcement.

As much as we hate to admit it, adults aren’t that different from children in this way.  Nobody takes well to being tricked into clicking on links or opening attachments.  Punishing them for it leads to even further resentment.  And where do you think they focus those hostilities?  The Security Team.  Those people who you are trying to protect end up blaming you and your team for getting them into trouble.  Now, what happens the next time you have a problem that you need that user’s assistance to solve?  Absolutely nothing.  Every time you phish a user, you are burning a bridge that you may need later on.  And since we all know how easy it is to phish a user, it just means that you are burning a lot of bridges.

So, what can we do to prevent our organization from being compromised by phishing and other types of social engineering attacks?  To start with, you should incorporate security awareness training to run alongside your new hire training activities.  Make sure that every employee has a baseline amount of knowledge on the issues and how to avoid them.  Next, you should invest in technologies that will detect and prevent these types of malicious activities.  Performing some sort of link and attachment inspection in e-mails and web content inspection for malware will significantly reduce the success rate of these types of attacks.  Lastly, there are a number of vendors who will track real-life phishing attempts to your users and modify the links to be able to perform analysis on who clicked and who didn’t.  This has the exact same effect of phishing your users, where you can sit them down and have a talk about what happened, but without pitting them against the Security Team.  The attacker is now the bad guy and you’re just the friendly information security professional helping to get them back up and running and giving them tips so that it doesn’t happen again.  You are BUILDING BRIDGES.  And, if you want to put an even more positive spin on this process, offer up a reward for those who get phished, but notify the Security Team instead of clicking on the link or opening the attachment.  Everybody wins.  That’s why you shouldn’t phish your users.

My First Six Months as an OWASP Board Member

When I first put my name in the hat for the OWASP elections in the fall of 2013, I thought I knew what I was signing up for.  I thought that my seven year history with the organization in a number of different roles (Chapter Leader, Chapter Committee Chair, AppSecUSA Chair) had me well prepared for the duties of an OWASP Board member.  I told my wife that it wouldn’t be a big deal, mostly something that I could do in my spare time while at work, and that it would feel good to be able to make a difference on a bigger scale than I’d done to date.  I ran for the Board on a platform of wanting to support the growth of the OWASP chapters around the world and wanting to drive visibility, and ultimately buy-in, back to the community.  I told myself that as passionate as I was with these things as a community member, it was time to either put up or shut up.

Here I am, six months later, as an elected member of the OWASP Board of Directors and I can honestly say that no prior experience could have prepared me for this.  It’s not a good thing or a bad thing, it’s just very different than I expected.  As a community member, I remember being at the AppSecUSA conferences and struggling with how to introduce myself to these “famous” OWASP Board Members.  I was a just a chapter leader struggling to come up with ideas to engage the Austin security community while these guys were literally trying to change the world.  They were the figurative “Rock Stars” of my little security world.  Needless to say, I see things a bit differently now, but it’s probably not what you think.

When I look at my fellow Board members, I do still see those “Rock Stars”.  I can’t even begin to tell you how much I look up to guys like Jim Manico for literally spending every day of his life trying to make the world more secure.  I constantly have to tell myself that even though I don’t consider myself a security rock star, the community saw something in me and put me on the Board for a reason and I continue to hold myself responsible for executing on the platform that I laid out in my election materials.  But what I’ve come to realize now, that I didn’t realize before my election, is that even though it feels the other way around, it’s really the community, not the Board that holds the power in OWASP.

When I look back at the discussions that we’ve had as a Board over the past six months, other than setting strategic goals, the vast majority of our meetings have focused on operational and governance issues.  Through this process, I have come to the realization that while extremely important to keeping OWASP, as a non-profit organization, afloat, this isn’t the kind of exciting world-wide impact stuff I thought I had signed up for.  As an example, my first two months as a Board member were spent in large part re-investigating a situation that a previous Board had closed the books on long ago.  In the process of trying to help the individual involved, I was twice accused by that individual (and acquitted) of violating OWASP’s Code of Ethics.  Talk about gratitude.  Since then, it seems like it’s been putting out one small fire after another.  More recently, I’ve spent many hours working with the Board and the Executive Director to grapple with an employee who resigned from the organization only to have members of our community question whether we, as an organization, did enough to keep them here, without knowing all of the details.  It blows my mind how the Board can have unanimous support for an item, feel confident that it’s in the best interest of the organization, and still be called into question as to whether we are somehow being underhanded in our decisions.  It’s like we sometimes forget that the Board is made up of seven people, from all over the world, with vastly different beliefs, desires, and even visions for OWASP.  If you can get that many people, that diverse, on the same page, then there’s something to be said for that.

So, I guess in a nutshell what I’m saying is that while I feel that it’s quite the privilege to be serving on the OWASP Board alongside some of the people I respect most in this industry, there is definitely a part of me that feels like the stuff that OWASP does that has the most profound impact on global security isn’t what we do on the Board, but rather, what the community does in our Chapters and Projects.  The Board is there to support you, the community.  To create the policies to make you successful.  To provide the staff to make your lives easier so that you can spend your time doing things that accomplish OWASP’s mission.  In addition, I want to dispel any notion that the Board is some sort of an Ivory Tower.  There should never be an “us vs them” mentality at OWASP because the Board is made up of people who have been, and in many cases still are, in the trenches right alongside the community.  The Board, to put it simply, is just a group of Chapter Leaders, Project Leaders, and other members of our community who, like me, decided that it was time to put up or shut up.  People who, for whatever reason, the community elected as our leaders to evangelize the OWASP mission and make the community that we hold near and dear to our hearts successful.  To think that anyone would volunteer to be a Board member only to destroy our community is absurd.  While I may not necessarily agree with everything my fellow Board members say or do, I have never questioned their loyalty to OWASP and I hope you don’t either.

With all of the above having been said, I feel that it’s also important to say that being an OWASP Board member is also an amazing opportunity to be a catalyst for change.  Over the past six months the Board has stepped up to the task of driving visibility and control back to our community.  We’ve instituted a new polling system that the Board have used to take the pulse of the community on key issues.  Michael has taken on the responsibility of weekly calls with the community in order to keep them informed of key issues and allow them to provide feedback.  And we are currently working on bringing back the committees under a new structure that will encourage participation and empower our leaders to take action.  OWASP even won the SC Magazine Editor’s Choice Award at this year’s RSA Conference.  Regardless of how you’ve felt about OWASP in the past, I feel quite strongly that the future for OWASP is so bright we’re going to need a good pair of shades.

So, I’ll end this post very similar to how it began.  The OWASP Foundation is currently accepting nominations for the OWASP Board of Directors.  If you’ve ever felt passionate about Information Security or felt like you have big ideas to make OWASP a better community, then now is the perfect time to throw your hat into the ring as I did.  I can’t promise that it’ll make you a security rock star.  I can’t even promise that the work is glamorous.  And my experience, thus far, has been that it’s been countless hours of volunteer work with little appreciation for what gets done.  But, what I can promise, is that OWASP is making the world a better place and the Board plays a vital role in making that happen.  You, too, can be a catalyst for change.