Observability vs. Monitoring in Snowflake: Understanding the Difference

Although used interchangeably, observability & monitoring describe two distinct data cloud processes. So if you’re investing in an observability vs. monitoring solution, it’s important to know the difference—otherwise you’ll have a hard time figuring out your return on investment. 

This article will not only define and clarify the differences between observability vs. monitoring, but explain the impact each of these processes can have on your Snowflake performance and spend. 

Observability vs. monitoring: what’s the difference? 

So what exactly is the difference between observability vs. monitoring? Both are key aspects of managing the data cloud, but they tell you very different pieces of information:

• Monitoring collects data within a system to find out whether the system is operating as expected. This process uses reports and alerts to identify and surface errors, faults, or anomalous data. 
• Observability incorporates additional situational and historical data into your analysis, providing the context necessary to identify root cause monitoring alerts. 

In other words, monitoring tells you there’s a problem, while observability tells you what caused it and why. 

As an IT concept, monitoring has been around since the dawn of the internet. In 1988, the creation of the Simple Network Management Protocol (SNMP) provided the first consistent IT monitoring standards. Later, OpenConfig and gNMI protocols allowed the first real-time monitoring capabilities, and is still used by many IT organizations today. 

Over time, however, IT infrastructures continued expanding and, eventually, companies needed more extensive visibility into those platforms. New capabilities like logs, metrics, and traces enabled deeper insight into why automated made the decisions they did, which is crucial to troubleshooting problems and coming up with solutions. 

Why observability is critical for Snowflake users

Snowflake observability is critical for anyone who wants to reduce costs and maximize performance efficiency. Both are dependent on usage patterns: a query runs longer than intended, a particular user or application has high demands, or your workload is unbalanced and consumes more resources than is efficient. 

For more information on why Snowflake usage directly impacts cost and performance, check out our Snowflake pricing guide. 

Monitoring may give you a heads up that a problem has occurred. For example, Snowflake provides resource monitors that send alerts and forcibly shut down warehouses when you hit a designated credit limit. As the name suggests, that’s a type of monitoring. But it does nothing to answer questions like the following: 

• If we hit our credit limit 30% faster in June vs. May, what caused that spike in spend? 
• Is the sudden uptick in usage a problem across the board, or is there one application or user that’s using up our compute? 
• Are our warehouses running too many queries for the resources provisioned to them, or are there a small number of inefficient queries that are using up our compute?

Snowflake’s resource monitors—and monitoring in general—won’t help you get to the root cause of your increase in spend. It only tells you that you’ve spent too much. Which, of course, is insufficient if you want to address the root problem. 

How does observability vs. monitoring work? 

The three pillars of observability: logs, metrics, & traces

Observability works off of three fundamental components: logs, metrics, & traces. Each of these is essential on its own, but also in terms of how they work together and talk to each other. 

Logs

In observability, logs are exactly what they sound like: structured or unstructured records of discrete, time-stamped events that take place within a particular system. Their fundamental purpose is to troubleshoot errors, identify anomalies, and audit activities for compliance. 

One of the major advantages of logs is their detail and granularity. However, this can present a challenge. Complex systems create high volumes of logs, which can seriously boost storage costs and make it overwhelming to parse them.

Metrics

Metrics are numerical representations of your system’s performance over time. These can include CPU usage, memory consumption, query latency, error rates, throughput, and more. 

There are a number of reasons why metrics are important, one of the biggest being the ability to take the data from logs and aggregate them into high-level views. This enables you to better monitor system performance, set up alerts for breaches of specified thresholds (e.g. error rates), and even analyze historical trends to inform and predict future behavior. 

Traces

Finally, traces enable you to track the journey of a single request or transaction as it moves along a distributed system. Traces include timestamps, service interactions, and error details to help visualize how your services interact. 

One of the key differences between observability and monitoring is that you can understand how and why specific events occurred, dependencies between services, and the reasons why certain components slow or fail. Traces provide you with the information needed to make that happen. 

How logs, metrics, & traces work together in Snowflake observability

So how do these three pillars of observability work together to provide insight into how your system behaves? 

Here’s a Snowflake-specific example: let’s consider an organization that’s had a high overall compute has been exceptionally high, causing the organization to consume all its Snowflake credits for the year halfway through Q3. 

• Metrics can look on a warehouse-by-warehouse basis and identify which ones are taking up the most compute
• Logs look at the individual queries executed by that warehouse to identify which ones used more compute power
• Traces help you figure out where those queries originated, and if they come from any particular user or application

Observability vs. monitoring differences

Monitoring, on the other hand, functions in a more straightforward way: tracking predetermined metrics that correlate to specific issues. Basically, for monitoring to work, you need to know in advance which metrics to track. 

With observability, on the other hand, you can rely on logs, metrics, and traces to identify root causes and, in some cases, flag potential issues before they post a problem. 

Final thoughts on observability vs. monitoring in Snowflake

From this article, it should be clear that if you want to prevent Snowflake costs from getting out of control or curb excessive drains on performance, you need an observability solution, not just monitoring. 

That’s where Keebo’s Snowflake Workload Intelligence comes in. This comprehensive FinOps & observability solution provides the features you need to track, investigate, and manage Snowflake performance and spend: 

• Real-time Snowflake insights. Keebo’s patented AI continuously monitors the health of your queries, warehouses, data, and storage to detect and diagnose problems.
• Actionable cost saving recommendations. Keebo analyzes millions of statistics to identify the actions that will have the greatest positive impact on savings & performance.
• Comprehensive, multidimensional optimization options. You have full control over where, when, and how you optimize. Choose from a variety of manual, human-approved, and fully autonomous actions.

Want to see the platform in action? Need advice on the best observability for your specific Snowflake needs? Contact our sales team today to get started.