Optimizing Snowflake Performance Is Difficult Because It Is Different
Snowflake provides a secure and scalable database engine with a unique architecture that decouples compute and storage resources, allowing you to pay only for the resources you use.
But there is no free lunch when it comes to optimizing your Snowflake.
- First, Snowflake is a newer offering compared to traditional data warehouses that have matured over decades. It has not yet provided all the fine-graining tuning tools that your DBAs typically use.
- Second, your Snowflake users are likely a broad mix of business and technical users, with varying levels of database proficiency, often resulting in poorly written queries, sub-optimal schemas, and inefficient data ingestion & consumption patterns.
- Third, Snowflake’s ease of scaling up/out has created a shortcut for solving performance problems allowing you to bypass best practice database optimizations at the cost of additional compute costs.
Cost Optimization Is Difficult and Different As Well
Snowflake cost optimization is also challenging due to Snowflake’s pricing model. Snowflake costs are usage-based, combining your use of storage, virtual warehouses (a.k.a. compute), cloud services, and serverless features. For most Snowflake customers, compute cost is by far the dominant factor, often an order of magnitude greater than storage and cloud services.
Ordinarily, the best way to cut compute costs is to optimize query performance – a win-win, so to speak. Unfortunately, optimizing individual queries won’t necessarily reduce your total compute costs. The reason is that Snowflake charges you a certain number of credits for every second that your warehouse is running, even if it is underutilized.
For example, consider an organization issuing 100K queries a month (larger organizations might issue tens of millions). Optimizing just 1% of the slowest queries would require its data engineers to inspect and improve a thousand queries—a daunting manual effort, far more than most organizations can execute.
What Snowflake Users Need to Optimize
Snowflake optimizations fall into four domains, each with unique tools and best practices. These include:
- Ingestion Optimization
- Query Optimization
- Schema Optimization
- Warehouse Optimization
These four types of optimization can be performed manually by your DBAs or data engineers or performed automatically. Saving you time, money, and engineering resources, automated optimization makes more sense.
How Does Automated Snowflake Optimization Work?
Explaining how automated Snowflake optimization works requires more specifics than this blog allows.
However, these details are readily available for you in Keebo’s Complete Guide to Optimizing Snowflake’s Cost and Query Performance.
This “must-read” for your Snowflake users and administrators helps your team understand Snowflake optimization challenges and solutions and provides you with actionable ways to cut your costs by 30-50% while also optimizing performance.
It all adds up.