Blog Post

Using Rocana’s Event Data Lake to Close SecOps Visibility Gaps

July 24, 2017 Author: William Kulju

Rocana Ops enables capture and analysis billions of events per day into a centralized event data lake, where all data is instantly available for online search for years. Because this event data lake can contain insights relevant to many different stakeholders, Rocana Ops almost always pulls double duty as a critical component of our customer’s security infrastructure. In this post, I’ll describe why and how Rocana Ops can help SecOps be more effective at discovering security threats and vulnerabilities.


Let’s start by imagining that you work in cyber forensics at a large enterprise. You’ve just received threat intelligence on a nasty new strain of malware starting to make the rounds. Armed with the indicators of compromise, you decide to proactively search applicable data sets for the telltale artifacts which will prove the hypothesis that your organization is already under attack.

There’s just one big problem: operational visibility gaps, which invariably arise for some or all of the following reasons:

1. You’re Being Inconsistent in the Data You Choose to Capture

It is highly unlikely your endpoint security monitoring infrastructure captures data consistently across the tens of thousands of servers, laptops, point-of-sale systems, and other endpoints in your environment. More data is captured from crown jewel systems, less from endpoints deemed low priority or third party managed systems. Some endpoints may not be monitored at all while others generate so much activity, the only way security monitoring tools can keep up is by filtering and discarding data aggressively. Whatever the reason, inconsistent data capture reduces the confidence you should have in your threat analysis. That’s because, even if your organization was exploited, you won’t find the forensic evidence in data that was never captured.

2. What You Do Capture, You Don’t Keep Long Enough

Security monitoring tools generally retain a few weeks or months of data at best. That’s OK because a best practice is to copy security data as quickly as possible to a tamper-proof central location for subsequent analysis and correlation with other critical data sets, including access logs, DNS logs, network traffic flow, email logs, and more.

But things get complicated when aggregate data volumes across sources exceed 10 TB/day or more as is typical in a large enterprise environment. For example, at these volumes SIEM ingest and index operations become serious bottlenecks even when assisted by the most performant (and expensive) hardware available. Faced with this constraint, organizations typically decide to retain data that can detect known threats and discard everything else. Except ‘everything else’ is precisely the data you need if you’re in cyber forensics searching for the telltale signs of a threat that wasn’t even known yesterday. Assuming the malware threat is already in your environment, you’ll never find evidence in data that was captured, but then discarded without being copied to your SIEM.

3. What You Do Retain, Grows Very Quickly

What data is ingested grows surprisingly fast. 10 TB ingested daily mushrooms into almost 1 PB of retained data every three months. Suddenly queries take hours or days to return results which means your ability to iteratively prove or disprove hypothesis and follow facts wherever they may lead is effectively gone. To your dismay, you learn the only way your SIEM administrator can maintain acceptable search performance is by discarding or offloading data to cold (and thus unsearchable) storage after a few days or weeks. And so, even though all the evidence you need to uncover the malware threat was in your SIEM, you are tragically no longer able to access that data.

After a long day of hunting, you found no evidence of the malware threat indicators. But you’re not celebrating because you know that proves nothing due to the huge gaps in the operational data that you are able to analyze. As is the case with an iceberg, just because you can’t see the threat below the surface, doesn’t mean it’s not there. To draw conclusions with confidence, you need to be able to start your analysis with a full picture of everything that happened in your environment during the timeframe in question.

As it turns out, you’re not alone in your need for total IT operational visibility to do your job better: your colleagues in IT Ops need it as well in order to maximize system uptime and stability. That is precisely the need that Rocana Ops was built to address, yet the core benefits of a centralized event data lake powered by Rocana Ops extend to SecOps. In particular, with Rocana Ops, SecOps teams can:

  • Have confidence that they are searching complete and full fidelity data. Rocana Ops all but eliminates the technical and cost challenges that have always discouraged CIOs and CISOs from capturing all data from all IT sources, including output from endpoint monitoring, threat intelligence, SIEM, and other security tools. Freed from these constraints, it’s been our experience that customers want to and do capture everything.
  • Search data at scale. Imagine how much faster you could pivot and prove or disprove hypothesis if you could search a trillion records and get answers back in seconds. Rocana Ops makes it possible with search performance that doesn’t degrade with data volumes or age.
  • Enjoy full fidelity data. With Rocana Ops, all data is stored “as captured” to preserve any evidence that you may be searching for. Any post-processing operations (e.g. resolving IP addresses) leave the original data intact and in a forensically sound, immutable state.
  • Provide real time curated data feeds in the expected format to any other tools in their arsenal. By offloading data capture, transformation, and enrichment operations to Rocana Ops, downstream analysis tools can operate far more efficiently.

If we now replay the above malware threat scenario but instead assume Rocana Ops is in the picture and being used to capture full fidelity data from across all endpoints and other sources, your search for evidence is likely to be much more fruitful. Moreover, if you don’t find any trace of the malware indicators of compromise, you can at least say with a high degree of confidence that your organization has not fallen victim to that particular threat. Either way, the underpinning by a Rocana Ops event data lake leads to a much more satisfying outcome for SecOps teams and companies everywhere.

Read the White Paper: Next-Gen Event Management

Learn About Rocana Ops: The Central Nervous System for IT Operations