Get the connector!
Effective data lake deployment requires one element that is often overlooked: deep understanding of the behavior and performance of each workload (set of queries that serves a business logic). Different workloads have different levels of impact on organizations’ business. Ranging from mission-critical, customer facing, internal dashboards, etc. Each workload, according to its priority, has its budget and the expected performance it requires.
Once organizations start to experiment with serving queries directly on the data lake, they are challenged with the transition from storage-only scaling to effectively serving critical workloads.
Most data platforms enable cluster monitoring of classic, technical metrics, that we know from the old “IT” world; metrics such as CPU load and RAM usage. This approach fails to align the business questions stream (workloads) and the TCO it generates.
Understanding the true cost of analytics requires expanding the classic “server monitoring approach” into a much deeper observability into the logic behind resource consumption, i.e. repeating query patterns, hidden dependencies, failure modes, cost and performance issues, etc.
Varada delivers deep and actionable workload-level observability that supports this transition to understanding how data is used by different workloads, how resources are allocated among different workloads and users, how and why bottlenecks occur, etc.
The workload perspective enables data teams to focus engineering efforts on meeting business requirements. As infrastructure bills heavily weigh on the momentum of analytics projects, even to the point of negative profitability, Varada delivers a much deeper understanding that empowers data teams to move the knob of cost-performance, such as spend, ratio vis-a-vis the actual business value or importance of the workload at hand. 
Workload-level observability empowers data teams to accurately evaluate whether the user experience meets requirements. Understanding workload bottlenecks allows data engineering efforts to focus on the most impactful actions and optimizations. Both the data and the nature of business user questions asked tend to change, sometimes abruptly. Without a deep monitoring, changes are often noticed and handled only after things break. By monitoring the workload behavior and identifying changes such as deteriorated execution times or inconsistent results, data teams can act preemptively and quickly adapt to changes.
The cost of data workloads is notoriously hard to predict and to keep under control. Workload-level observability enables identifying which query patterns consume resources ineffectively and drive the cost of a workload. Armed with this information, data teams can better predict the spend on a workload and focus their cost reduction efforts where they matter the most. Furthermore, workload-level observability enables data teams to prioritize workloads according to their business impact and focus their price-performance optimizations accordingly.
Many analytic processes are iterative by nature. With business objectives in mind, analysts map the realm of data to extract relevant insights, verify them and understand their context and impact. Being able to ask more questions, and being able to ask them interactively without interrupting the analysis flow, leads to better understanding of the data. Strong visibility of the workload enables data teams to accurately evaluate and optimize to deliver this sought-after user experience.
