In my previous blog post, I shared some of my thoughts about a recent article by Matt Asay at InfoWorld, titled “Why Splunk keeps beating open source competitors.” I provided examples of how the market is shifting toward adopting centralized event warehouses built on open source components, and how this led to the creation of Rocana Ops.
In this blog post, I explore the situations that lead people to misuse monitoring tools like Splunk; how the misuse of tools stunts organizational changes that need to occur in order to successfully drive Digital Transformation; and what the path forward looks like for adoption of data warehouses for IT underpinned by open source.
Breaking Conway’s Law
We recognize that companies with significant Splunk investments aren’t looking to rip and replace solutions that work just because they’re more open. Those familiar with Conway’s Law know that as organizations design systems, the result is systems that are a copy of the organization’s communication structure. As Intellyx analyst Jason Bloomberg wrote on our blog back in November, Conway, a software developer, “was observing that siloed development teams build siloed software systems. The reverse also holds: siloed software systems reinforce organizational silos.”
How is this relevant to open source and Splunk? Matt cites Box engineer Jeff Weinstein's assessment that Splunk's continued adoption is driven by "misuse". Enterprises are "pushing data into Splunk for jobs it may not be particularly well-suited to manage." While most organizations need a centralized event warehouse, IT teams are building it with hammers when the appropriate tool looks more like a massive 3D printer. The market isn’t just looking for a version of Splunk built using open source tools, it’s looking for the next evolution of monitoring: total operational visibility.
Use the Right Tool for the Right Job
Consider just one example of misuse we see frequently: security forensics teams with large Splunk investments. Security tools like Splunk support collecting pre-identified security log sources and then applying correlation rules and saved searches. As the demand for 100% visibility has increased – due to compliance and incident response requirements – traditional security solutions have struggled to adapt because of scale challenges, lack of advanced analytics and machine learning, cost of hardware for long term online retention, and onerous software licensing.
Recognizing this demand and struggle, we developed Rocana Ops to solve both the scale and cost issues presented by traditional tools while adding the advanced analytics capabilities and also integrating with legacy tools so teams can continue to use them where they shine.
With Rocana Ops, organizations are collecting 100% of their event data from logs, metrics, network and application instrumentation, and continue using existing workflows without overwhelming legacy tools. The event data is stored, analyzed and visualized using a single interface alongside other enterprise event sources, such as application, database, web, and middleware systems. Rocana Ops gives operators the ability to see the full context of all event information resulting in faster and more accurate responses. Meanwhile, just the relevant events are forwarded to Splunk and other applications for processing by existing correlation rules and saves searches, reducing overall costs and dramatically increasing IT operations visibility and capabilities.
Prepare for Change
Ultimately, we continue to see evidence supporting Matt’s original thesis. Open source has created the opportunity for organizations to wean themselves off of legacy tools like Splunk, provided the right packaging and support. More importantly, we’re finding that open source is the primary contender to supplant Splunk at large data volumes. The path there starts with helping organizations keep Splunk and other tools for their original needs and look to next generation technologies like Rocana to be the central nervous system of operational data, feeding downstream tools like Splunk for analysis and visualization.
The data warehouse opportunity for IT is untapped. IT desperately needs a centralized warehouse for all event data. That market is ripe for open source-based solutions to dominate. And this is where we believe Rocana Ops with its blend of homegrown software built on an open source framework will lead the way.