Expanding Operational Data to Answer Customer Demand

By Ori Reshef
December 3, 2020
December 3, 2020

Industry: Cyber Security (Endpoint Protection)

Challenge: Customers require access to extensive data at their fingertips

Varada’s customer, a leading data-driven endpoint protection vendor, monitors 100B+ endpoint events every month for thousands of enterprise customers. By delivering up-to-date and accurate information on endpoint threats at the fingertips of customers, this vendor is able to provide unique insight to prevent, detect, respond, and hunt attacks across all enterprise assets.

The customer encountered significant challenges in fully leveraging the value of its rapidly growing and unique data and meeting tight customer-facing data access SLAs.

“Data is at the heart of our offering but we couldn’t find a solution that enabled us to leverage the huge amounts of data we collect. We had to compromise: it was either exposing data at scale, providing customers valuable insights on various dimensions, or meeting performance and concurrency SLAs.”

This compromise meant that in order to deliver interactive performance for end-customers, only 7 days of data were exposed. De facto, thought massive amounts of data were collected, only a small portion was “actionable” and monetized. In addition, to ensure fast response time, end-customers could not leverage the platform to drill down and deepen insights.

Solution: Fast and efficient indexed data tier to serve applications, on top of the data lake

To eliminate this compromise and unlock the value of its data, this customer turned to Varada. Varada’s platform leverages the existing data lake, as a single source of truth. Deployed within the customer’s VPC, Varada seamlessly connects to the existing data lake and creates an Indexed View as a “hot data” tier between the data lake and customer-facing SQL applications.

Varada Platform Overview

“By keeping our existing data lake and operating within our own VPC, Varada’s solution was not only easy to deploy, it also enabled us to apply our own strict security and data policies and maintain full control throughout the data and application stack”.

Varada also combines last-mile ETL which enabled the customer to easily define the scale of the Indexed View. One of the customer’s main goals was to expand its operational data set from 7 days to 90 days. The team was easily able to achieve that with a simple SQL command that defined this “moving window” definition of the Indexed View.

Unlike pre-aggregations or projections used for operational analytic needs, Varada’s Indexed View maintains the full cardinality and dimensionality of the underlying data, ensuring maximum flexibility and eliminating the need to predict and optimize for specific user workflows. To ensure constant data freshness, Varada’s Indexed View is automatically synchronized with the data lake.

Bottom line, Varada enables this customer to serve SLA-sensitive applications on massive amounts of data:

  • Easily define any operational dataset on any data source, and use standard SQL
  • Analyze data across any dimension and at any granularity
  • Ensure data freshness
  • Cut down on manual ETL efforts
  • Keep data in your full control, in your VPC

Results: Very fast queries on significantly larger operational data set

x10 increase in operational dataset.

x100 faster response time for advanced selective searches.

x10 less resources required, based on optimized cluster management and simplified data flow.

Significantly expanding both the operational dataset exposed to its customers, and the granularity and sophistication of possible searches, enabled the customer to boost its value proposition to enterprise customers.
This increase in the scale of data exposed to customers, from a 7-day view to a 90-day view, did not require any performance compromises.

“Based on Varada’s platform, we were able to run complex searches while simplifying the data flow. We improved query response time by 100 times for exact match filter on events and by 10-20 times for JOIN queries on data and events”.

By leveraging Varada’s highly efficient inline indexing, which is optimized for i3 machines and NVMe SSD, the customer was able to reduce its AWS resource usage on r5 instances for analytics by x10, boosting the cost performance moving forward.

We use cookies to improve your experience. To learn more, please see our Privacy Policy