This post was written by Brad Freeman, SenseOn’s Director of Technology. Brad has over 15 years of experience in conducting national cyber security investigations across critical national infrastructure and telecommunications sectors. He has led threat hunting teams at corporations such as BT, managed Security Operations at EE as well as performing incident response on offshore Oil & Gas platforms. Drawing upon his extensive experience and knowledge, including CISSP & CISM certifications, Brad leads the threat analytics team at SenseOn. He applies ML and AI to automate the process of detecting and investigating cyber adversaries, overall specialises in uncovering advanced actors within customer environments.
Before becoming SenseOn’s co-founder and Director of Technology, I spent over a decade managing security teams at various organisations. Although I relished the challenge of providing frontline security across different verticals, these experiences were also connected by a common negative thread.
No matter how much we invested in defence, there seemed to be an immense gap between what we wanted our security solutions to do and what they actually did. In no instance was this more true than in how we used SIEM.
Working as security operations manager for an international energy company might be what first made me acutely aware of the downsides of SIEM tools.
With a lot to lose from even a minor cyber attack, my organisation was keen to invest in cybersecurity. And, after a multi-year, several million-pound project to install a SIEM solution purchased from a household name vendor, my ten-person security team appeared to be well equipped to deal with advanced threats — at least at first glance.
Thanks to SIEM, my team would receive real-time event logs from tens of thousands of servers across our entire IT estate. Essentially, whenever anything remotely suspicious happened, we would know about it.
Unfortunately, while this might have sounded good in theory, receiving thousands of logs every day created a critical problem — how do you find business value among what turned out to be an immense amount of noise? As a result, the first thing we had to figure out was how to manage the overwhelming volume of alerts that our SIEM solution sent us without compromising security.
We spent countless hours trying to solve this problem. Working closely with our security engineering team, I spent most of my days writing “playbooks” on how to configure technical controls in a way that gave us value. To try and offload some of the stress SIEM was putting on experienced analysts, we created flow charts that would tell junior security analysts precisely what to do in case of a particular alert or sequence of alerts. The hope here was that senior analysts would then be freed up to do threat intelligence.
Tied into our SIEM solution, which categorised ten types of alerts, this security framework seemed like a sensible way of mitigating the stress that came from having to deal with a barrage of alerts on a daily basis.
Unfortunately, trying to extract value from a SIEM solution only proved to feed into an increasingly circular problem.
We soon discovered that a SIEM is utterly unsuitable for real-world security environments, even with well-thought-out orchestration and planning. For example, to detect brute force attacks, I configured our SIEM to alert us every time a user failed to enter a correct password more than a certain number of times each minute.
Unfortunately, after doing so, my team soon received an extremely high number of alerts related to failed password entries. Our SIEM seemed to indicate that we were suffering from regular brute force attacks but gave us no context. We also couldn’t find any other evidence of compromise. This meant that each time we received an alert, a member of my team had to take valuable time away from investigating other threat alerts to ask staff members questions like, “when did you last change your password?” Meanwhile, our SIEM would send us more alerts. As the backlog of tickets mounted, our capacity to investigate threats further decreased.
After a prolonged investigation, we discovered that certain programs like Skype would keep a user’s old password in the background. Without users themselves doing anything, these programs would end up re-trying old passwords every couple of milliseconds, resulting in hundreds of failed login attempts every time an employee changed their password. However, by the time we figured this out, our SIEM had encouraged us to waste a considerable amount of time searching for compromises that it had invented.
A similar situation would happen whenever an employee left the business. For whatever reason, their devices were often still plugged into our network after their departure. This resulted in our network constantly receiving connections from ex-employees trying to authenticate with the network. Although this was likely because individuals had hardcoded their credentials or logged into a server and kept the session alive, our SIEM gave us no insight into why this was happening. All it did was send us siloed alerts, forcing us to investigate non-existent problems.
To try and stem this flow of false positive alarms, of which the above are only two examples that come to mind, I spent much of my time going back and forth with our security engineering department. We would repeatedly try to figure out where these alerts came from and engineer them away — only for more alerts to pop up in their place. I lost count of how many times I said, “this use case is bad. Make changes here.”
As well as creating false positives and causing 46% of operational downtime, the fundamental flaw with SIEM solutions is that whatever data they provide is completely siloed.
They don’t give security teams enough context to get to the root cause of alerts or queries quickly. When the National Cyber Security Center (NCSC) published a report saying that sponsored state threat actors were using specific IP addresses to attack an organisation I was working with, this became an urgent issue for my team and me.
To figure out whether my organisation was being targeted, I entered a selection of the known threat actor IP addresses provided by the NCSC into my company’s SIEM. Shockingly, my SIEM solution reported that it found a thousand web requests to one of those IP addresses in the course of a day — all from a single device.
The high frequency of connections immediately made me think that I had uncovered command and control traffic (C2). Naturally, with patterns that match C2 going to what the NCSC has said is a nation-state threat actor, I panicked. Along with my best security analyst, we rushed to investigate the device where the traffic was coming from. However, we didn’t find any evidence that the computer was compromised. Of course, because advanced threat actors can move laterally without leaving traces, we still had no idea whether or not the device had been previously compromised.
Eventually, after days of focused, high-pressure investigation, we discovered that because the IP address that the NCIC highlighted was on a shared IP range, it shared its IP with over a thousand websites. One of our employees had used one of these sites to refresh sports results constantly. Thus, the siloed information that SIEM provided indicated an employee’s sports obsession, not C2.
The above examples are only a few of the times I can think of when out-of-context SIEM alerts led my team and me down a garden path — taking us away from our actual jobs, creating vast amounts of stress, and, ultimately, hurting real security.
This happened to me, and, unfortunately, it still happens to security teams daily. The reason why is that a SIEM tool works like an email inbox. Alerts come through, and it’s left up to you and your team to deal with them individually. I used to describe SIEM as “Excel on steroids” because, at the end of the day, although SIEM can give you endless logs, you still have to write your own rules. Out of the box, SIEM doesn’t do much.
We built SenseOn to succeed exactly where SIEM fails. Our self-driving cyber defence platform goes beyond the limitations of SIEM, improving visibility and security while lightening the load for security teams. SenseOn clusters disparate security observations to find real compromises. A single pane of glass also gives security teams a rich picture of alerts and vulnerabilities.
As a tool for threat investigation, SenseOn shows analysts precisely where alerts are coming from at the user and application levels. This means that situations like those I described above never happen.
Unlike SIEM tools, SenseOn doesn’t wake you at 3 AM for false alarms or send you hundreds of meaningless alerts each day. What SenseOn does is secure your entire IT estate, from endpoints to cloud servers, automating routine investigation while providing your analysts with data-rich insights into real threats.
Explore our collection of eBooks, webinars, articles, and more to help you maximize your understanding of emerging threats, adversary techniques and how to detect cyber attacks.Visit resource hub
Join thousands of like-minded professionals who are already receiving our blog updates and best practice guides.