Automated Data Cloud Optimization for Fintech: It’s Less Risky than You Think

Over 90% of Fintech companies use AI, mostly to aggregate and summarize information, but not to automate decision-making. That’s left to human operators. While helpful in many use cases, this approach is limited in its scalability and presents a challenge for Fintech data cloud optimization efforts.  

Data cloud optimization isn’t a one-and-done deal. Demands on warehouses are always in flux. For a human operator to achieve maximum optimization, they would have to make adjustments to every warehouse every minute, every hour, every day. A better approach is to use an AI to automate these adjustments in real time. 

But for Fintech, when real dollars and highly sensitive information is on the line, there can be some risk involved in AI-powered optimization. People are understandably gun-shy about it. However, there’s an equal (perhaps greater) risk of handling your optimizations manually—higher cost, slower performance, and lost opportunities due to resource misallocation. 

So how do you automated Fintech data cloud optimization without incurring undue risk? Read on for our solution to this problem. 

The pressing need for cost optimization in Fintech solutions

Fintech funding remains tight in 2024, especially when compared with high rates of investment just a few years prior. Combine that with ongoing inflationary pressures and broader decline in access to capital, and Fintech companies find themselves needing to do more with less. 

When it comes to data warehousing costs, however, cutting back on expenses is easier said than done. Fintech software solutions manage highly sensitive data that requires extra security and privacy. The cost of managing those data, then, isn’t limited to infrastructure, but includes:

• Software licenses beyond the data infrastructure platform itself
• Data processing fees, which increase as your user base grows
• Governance, security, and compliance expenses—which is mission critical for Fintech especially
• Personnel training to ensure good management of the data

This has resulted in an unprecedented investment in FinOps training so engineers can implement best practices for cost control. The problem is that most Fintech companies are already strapped for cash. Adding an entirely new business function—or diverting resources away from shipping product—is unfeasible. 

At the same time, Fintech companies can’t afford to ignore cost optimization solutions that can potentially free up valuable resources. So you end up in a catch-22: you can’t afford to optimize, but neither can you afford not to optimize.  

Why Fintech cost optimization solutions must be automated

So how do you get out of this proverbial “rock-and-hard-place”? The fundamental problem is the conventional approach to cloud cost optimization: namely, relying on human engineers to make and implement optimization decisions. Here’s why that’s not the most effective way to go. 

Dynamic cloud environments

Data demands in cloud environments are never completely static. So while you could theoretically configure Snowflake based on today’s ideal state, as time marches on you have to constantly keep up with the shifting demands on your warehouses. 

Any number of changes can impact the number of resources you need to provision to manage your data cloud workloads:

• Fluctuating user counts
• Variable workloads among those users
• Variable query complexity (more complex queries = more resources)

These changes can happen at any time of day. There’s no predicting when or where they’ll occur. Which means it’s not feasible to have a team of DBAs on call 24/7 to upscale warehouses for a ten-minute workload spike, then downscale once the workload reverts to normal levels. 

However, making these small, real-time adjustments are critical to optimize your costs. If you always run workloads on small warehouses, your queries will run for longer and potentially burn more credits. If you always run them on large warehouses, you’ll burn through credits faster than you need to. 

Distributed business logic

Another common problem facing cloud cost optimization is distributed business logic. Ideally, you’d organize your warehouses so that small warehouses run small workloads, medium warehouses medium workloads, and large warehouses large workloads. 

In reality, however, business logic results in queries distributed across your data warehouses in ways that may make sense based on the structure of your application. For example: 

• Service-level agreements require certain resources to be available for certain use cases, often with performance expectations built in
• Some enterprise clients may have dedicated warehouses for their data
• Specific tools or platforms may rely on a specific workload—this is especially common with BI tools like Tableau, Looker, Sigma, etc. 

Maintaining and optimizing query logic becomes extremely challenging when distributed across multiple ETL jobs, pipeline definitions, and application business logic. Routing queries to the appropriate warehouse is impossible for a human engineer to stay on top of 24/7. 

Necessary data expertise to implement

Ask yourself another question: is constantly adjusting Snowflake the best use of your engineers’ and DBAs’ time? Because predicting, scaling, and managing warehouses requires more expertise than an entry-level team member has. 

You don’t want to pay your engineers to constantly change warehouse sizes. You want them building new data pipelines, managing your database, maintaining overall system security, etc. Optimization, while important, can’t become their full-time focus. They’re too valuable for that. 

How to mitigate the risks of automated, AI-powered cloud data optimization

The reality is that if you want to optimize your cloud costs, you need an AI-powered solution. And that’s where things get a bit tricky. The sensitivity of financial data and performance demands from clients make it hard for Fintech companies to hand these optimizations over to an AI. 

Thankfully, there are three aspects of Keebo’s approach that help to avoid these risks. Let’s briefly take a look at each. 

1. Predictive resource management

Keebo’s AI-powered cloud optimization tool analyzes activity logs, performance metrics, usage patterns, and other historical data to identify real trends in data cloud usage. From there, we’re able to forecast future demands not on industry-wide benchmarks, but real usage. 

Our platform accomplishes this through reinforcement learning, an AI technique that mimics the human decision-making process. By starting with a stated goal or outcome, RL rewards only those actions that help to deliver that outcome. 

Because our AI tools are constantly working in real cloud environments, our automated adjustments are more accurate than virtually any competitor. 

2. Performance guardrails

If you have specific performance requirements in place, whether formal or informal, you can use Keebo’s performance guardrails to protect your performance. When an optimization results in a long queue time, high number of queued queries, or extended query latency, Keebo will back off its optimizations and let your warehouse default to its larger size. 

3. Metadata only

Not only is Keebo secure (we’re 100% SOC2 and GDPR compliant) but we also use zero customer data in our models. Our AI tool only looks at Snowflake metadata: data structure, query logs, execution times, usage statistics, and more. So you never have to worry about your data falling into the wrong hands. 

Learn more about Keebo’s AI-powered optimization tools for Fintech companies here.