About two years a sales rep from a company called Verkada reached out to me to evaluate their approach to physical security monitoring for our campus. While our physical security isn’t under my team’s purview, it’s something I’ve always been interested in and we’ve helped to advise the facilities team responsible for it in the past. I decided to take them up on their offer to demo one of their cameras.

The device itself looked pretty cool. It appeared to be a high quality camera with capabilities to zoom in and focus and it came with hardware to mount it outside. The place I identified as an ideal location to mount it was high up above my driveway on the front side of the house and I hired someone to come out and run the wiring for it. A single ethernet cable with power running through it as well.

Once I got it up and running, I was very impressed. It was a decent quality camera that was constantly recording video to the cloud and allowed me to easily hone in on events of interest. Things like when my brother-in-law invited a girl over to our house while we were away on vacation or the kids who stole our bowl of Halloween candy. For over a year and a half it did exactly as advertised and I sang it’s praises to many people because of that. Then, this past April, the camera stopped recording and all I saw was a message saying it was updating.

I waited for days…weeks…over a month for the update to complete. I tried rebooting it several times as the Verkada website talks about how they keep a backup copy of the firmware so the device is virtually unbrickable. Nothing worked so eventually I reached out to Verkada support.

After some light troubleshooting around the color of the lights on the camera and what they were doing, support blamed it on my firewall. They said that it was “internet access related”. Must be the firewall blocking access to their cloud endpoints, NTP, or DNS resolution. The finger pointing had begun.

If you’ve read any of my blog posts, you probably know that I’m no dummy. I had already been reviewing my firewall configuration and logs and had determined that the DNS traffic was going through just fine and the only oddity was that the client side of the connection it was making to their servers showed an odd “client-rst” action. They asked for a packet capture, which I sent to them, and concluded “the firewall or something upstream is doing some SSL inspection which is affecting the connection”. I had created an isolated policy to help troubleshoot the issue that had no rules other than “allow all ports and protocols from this device to any location”. They claimed that “on the backend we are not seeing successful connections from the camera in regards to data contained in the TLS tunnel” (something that turned out to be outright made up because I showed responses from their servers in the packets). They said “this looks like the firewall is blocking the camera from making a TLS connection to the cloud”.

All of my troubleshooting at this point pointed to issues with the camera closing the connection, but they continued to point the finger at my firewall so I submitted a support ticket with Fortinet support. Fortinet support responds with “Looking at the output there are no drops on this capture. The firewall has distinct messages that it will provide when it drops a packet, indicating why. The example traffic you sent me shows the 3 way handshake completing before the Reset Packet originates from [the device]… Based on the debug flow, the Firewall is not dropping the traffic, and the traffic based on the successful complete of the 3 way handshake is flowing in both directions and received by all the involved devices… After the completion of 3 way handshake we are seeing the Reset coming in from the Client device. The Packet comes in as a Reset from the Client, on port2. Note that if the Fortigate was intervening and sending the reset on the Clients behalf it would show it originating from “Local” instead of Port2.”

At this point, I figured out how to turn off ASIC offloading on the firewall and got a full PCAP of what was going on. I noted a ton of “Certificate Expired” messages each time the client tried to negotiate a TLS connection with the server. Several more PCAPs later and the Verkada support person eventually tells me “I believe the camera may be experiencing a bug with system that affected a small population of cameras in the wild. We have since patched this issue but on rare occasions a camera cannot be recovered.” At this point I’ve spent several days troubleshooting an issue that they knew about, but instead pointed fingers at my firewall vendor. I’m kinda upset about how we got here, but at least we’re at the point where they’re going to RMA it and send me a device that works properly, right? Nope.

The following day I get a final email from the Verkada support representative telling me “I submitted an RMA but it was denied. The reason being, the camera was a beta giveaway and was never paid for.” I get it, I didn’t pay for it, and I can respect their desire to not want to pay to ship me another one. That being said, I accepted this under the pretense that Verkada was providing me with a unit to evaluate and provide feedback to my facilities team and others in the InfoSec community. I provided positive feedback when it was up and running, but now that it’s gone belly up I feel like I need to share that as a data point for perspective customers, now, as well.

I guess that one could argue that a customer would have paid for the device and the eventual RMA would have resulted in a new one being sent, but these aren’t being sold to techies like me. These are being sold to businesses who want an easy way to monitor what is going on around them. Businesses who don’t have the time or ability to send PCAP after PCAP, the expertise to push back against the finger pointing by support, and the ability to involve support from their firewall vendor to get to that resolution. And even if the business got to the RMA, do they really want to have to go through the hassle of pulling all of the devices down and putting new ones up?

At this point, I’ve lost all faith in Verkada’s product and their ability to properly support enterprise customers. I rescind all recommendations that I’ve made for them in the past and would highly encourage others to look elsewhere for their IP camera purchases.