← All articles

The Half-Million Dollar AWS Diet: Grabyo’s Cost-Cutting Success Story — Part I

Introduction

At Grabyo, our engineering team thrives on innovation, constantly pushing the boundaries with cutting-edge technologies. As the head of the Platform team, my focus has traditionally centered on elevating developer experience and maintaining ironclad security. However, the landscape shifted dramatically in early 2023 when our CEO initiated a company-wide cost optimization drive.

This directive prompted a strategic pivot. While continuing our efforts in developer experience and security, we expanded our scope to include aggressive cost optimization. Over the subsequent months, we meticulously dissected our infrastructure, hunting for inefficiencies and opportunities to streamline.

The results were remarkable: we managed to slash Grabyo’s annual AWS spend by nearly half a million pounds. This post, the first in a two-part series, details the initial phase of our journey, sharing the strategies employed, challenges encountered, and key insights gained.

Our Approach: Leveraging AWS Cost Explorer for Strategic Insights

Our cost-saving odyssey began with a deep dive into AWS Cost Explorer. This powerful tool provided crucial insights into our spending patterns across all AWS accounts and services.

The approach was methodical:

1. Analyze utilization of every AWS service
2. Prioritize high-impact areas with the most significant saving potential
3. Examine expenditure history over the past few years
4. Identify anomalies where spending didn’t align with platform growth

This process uncovered both justified cost increases (like higher network traffic due to enhanced security measures) and unwarranted spikes (such as a malfunctioning content deletion system leading to unnecessary S3 storage costs).

Our Cost-Saving Strategy: A Four-Pronged Approach

Our journey to optimize AWS costs was guided by four key strategies:

• Enhancing system performance: We focused on improving the efficiency of our existing systems, allowing us to do more with less.
• Optimizing resource allocation: We identified and eliminated waste, ensuring our resources were used effectively.
• Strategic service migrations: We explored alternative services that could provide similar or better functionality at a lower cost.
• Retiring and consolidating services: We critically evaluated our services, retiring those that were no longer providing value and consolidating where possible.

These strategies formed the backbone of our decision-making process, helping us prioritize our efforts and maximize our cost savings. In the following sections, we’ll dive into specific initiatives that exemplify each of these strategies.

Performance Enhancements and Refinements

Live Encoder Performance Tuning: $320,000

At Grabyo, live encoding forms the core of our operations, consuming a significant portion of our infrastructure budget. When we embarked on our cost-saving journey, it was the obvious place to start. This initiative not only yielded the most substantial savings but was also surprisingly straightforward to implement, slashing our EC2 costs by $320,000 per year — accounting for nearly 70% of our total cost savings.

We’ve always prioritized keeping our live encoding fleet current with the latest instance families. Newer generations typically offer improved performance at a slightly lower cost, providing consistent benefits in efficiency and economy. During our most recent instance family upgrade, however, we discovered something extraordinary.

Beyond the expected minor cost reduction, we observed a significant performance increase. This boost was substantial enough to allow us to reduce the instance size of our live encoders by half without compromising on quality or capabilities. It was an unexpected windfall, turning a routine upgrade into a major cost-saving opportunity.

While we’re excited to share our success, we also need to protect our competitive edge. The specific instance families and sizes we use are closely guarded secrets — they’re key to Grabyo’s efficient live encoding. But the principle remains: always be on the lookout for opportunities to optimize, even in areas where you think you’ve already maximized efficiency.

Fargate Rightsizing: $14,400

Our cost optimization efforts revealed an unexpected issue within our AWS Fargate-based microservices: systemic over-provisioning caused by our own microservices template.

The template, designed to accelerate development with boilerplate code, included overly generous default resource allocations. This “one-size-fits-all” approach ensured smooth operation but led to excessive unused capacity across many services.

We conducted a thorough review of our Fargate-based services, analyzing CPU and memory usage patterns. Through careful optimization, we right-sized our Fargate tasks to match each service’s actual requirements while maintaining a safety buffer.

Key outcomes:

• Reduced AWS costs by $1,200 monthly ($14,400 annually)
• Refined our microservices template with more appropriate default allocations
• Established a new best practice: running databases and services within a 50% to 70% utilization range

The Fargate performance tuning initiative reminds us that optimizing cloud costs often involves fine-tuning small, everyday defaults that can have a large cumulative impact.

Database Architecture Rethinking: $10,140

In the world of microservices, the principle of “one service, one database” is often treated as gospel. At Grabyo, we had diligently followed this rule across our infrastructure, encompassing ElastiCache for caching, RDS for microservices databases, and OpenSearch for analytics. While this approach promised perfect isolation, it led us down a path of increasing inefficiency and complexity.

As our platform grew, so did our challenges. We found ourselves managing a sprawling landscape of underutilized databases, each incurring full costs despite often minimal usage. The overhead of maintaining these numerous instances became a significant burden on our team. Perhaps most concerning, the high cost and effort required to deploy new services with dedicated databases began to stifle our ability to innovate and scale efficiently.

It was clear that a change was needed, but the solution wasn’t immediately obvious. How could we optimize without sacrificing the benefits of isolation? The answer came in the form of a comprehensive strategy that would transform our database infrastructure.

Our approach centered on consolidation across our entire database ecosystem. We merged all databases of each type into a single cluster, leveraging built-in auto-scaling for ElastiCache and OpenSearch. The game-changer was our approach to RDS: we transitioned all microservices to a shared Aurora Serverless cluster. Aurora Serverless’s automatic scaling mitigated the typical challenges of shared databases, giving us confidence in our consolidation strategy. For ElastiCache and OpenSearch, we also migrated to AWS’s ARM-based Graviton processors, yielding a 20% cost reduction while maintaining or improving performance. Finally, we enhanced security by relocating all database services to private subnets, eliminating the need for public Elastic IPs and further reducing costs.

The results of this unified approach were remarkable:

• ElastiCache optimization: $4,880 annual savings
• Microservices database consolidation (RDS): $1,600 annual savings
• OpenSearch consolidation: $3660 annual savings

Beyond the financial benefits, we saw dramatic improvements in resource utilization and scalability, particularly with Aurora Serverless adeptly handling our microservices’ varying workloads.

Perhaps the most significant outcome was the removal of barriers to innovation. No longer did our teams need to justify the high costs of dedicated databases for new services. This newfound flexibility accelerated our development cycles and allowed us to respond more nimbly to market demands.

GP2 to GP3 Migration: $5,475

As part of our ongoing efforts to scrutinize every aspect of our infrastructure, we turned our attention to our EC2 instances’ storage. We realized that all our EC2 servers were using General Purpose SSD (gp2) volumes, which had been the standard choice for many years. However, AWS had recently introduced a new generation of General Purpose SSD volumes: gp3.

The gp3 volumes offered a compelling proposition — significantly lower costs without compromising on performance. In fact, in many cases, gp3 volumes provide better performance than their gp2 counterparts.

We embarked on a straightforward yet highly effective initiative: migrating all our EC2 servers from gp2 to gp3 volumes. The process was relatively simple, involving minimal downtime and risk.

The results? This seemingly small change led to an impressive annual saving of $5,475. It’s a perfect example of how staying informed about the latest AWS offerings and being willing to make incremental changes can lead to substantial cost reductions over time.

Resource Optimization and Waste Reduction

EC2 Cost Detective Implementation: $27,600

At Grabyo, our extensive video transcoding operations necessitate significant EC2 usage across our infrastructure. This high utilization isn’t limited to our production environments; it extends to our development and sales processes as well. Our engineering team relies on EC2 encoders for development and testing, while our sales and marketing teams frequently spin up instances to demonstrate our platform to potential clients.

To keep our operational costs in check, we’ve always encouraged both teams to shut down these non-production instances when they’re no longer needed. However, relying on manual shutdown processes was far from foolproof. Instances were often left running unnecessarily, silently accumulating costs.

Enter the EC2 Cost Detective — our automated solution to this persistent problem. This service monitors EC2 usage across all our regions and accounts, focusing on non-client instances. When it detects a server that’s been running for over 8 hours without a valid reason (i.e., not associated with a real client), it sends an alert to our team via Slack.

The implementation of EC2 Cost Detective yielded immediate and substantial results. We quickly identified and shut down numerous forgotten instances, leading to direct savings of approximately $27,600 per year in unnecessary EC2 costs.

However, the true value of this initiative extends beyond these quantifiable savings. The proactive nature of our alerting system has prevented countless instances from running unnecessarily for extended periods. While it’s challenging to put an exact figure on these preventive savings, we estimate they significantly surpass our directly measurable cost reductions.

Moreover, EC2 Cost Detective has fostered a culture of cost-awareness among our teams. The regular Slack notifications serve as constant reminders of the impact of our infrastructure usage, encouraging more mindful resource management across the organization.

AMI & Docker Cost Detective: $21,612

At Grabyo, our commitment to immutable infrastructure has long been a cornerstone of our deployment strategy. Each software release spawns a new Amazon Machine Image (AMI) for our EC2 instances and a new Docker image for our containerized services. This approach facilitates our blue/green deployments and ensures consistent, reliable infrastructure. However, it inadvertently led to significant, and initially overlooked, cost centers.

The problem emerged gradually. As our release frequency increased, so did our AMI and Docker image production rates. Coupled with the cumulative effect of years of retention, we found ourselves facing mounting AMI and Docker image storage costs. The root cause? A lack of structured lifecycle management for both AMIs and Docker images in our systems.

To tackle this issue, we developed the AMI & Docker Cost Detective. This tool operates by:

1. Scanning all AMIs and Docker images across our AWS accounts
2. Cross-referencing AMIs with existing launch templates and launch configurations, and Docker images with task definitions in Fargate
3. Identifying resources that are:
   a) Not referenced by any active configurations or task definitions
   b) Older than three months, while preserving the latest 10 versions
4. Automatically removing the identified AMIs and Docker images

The implementation of AMI & Docker Cost Detective yielded immediate and substantial results. The total annual savings from this initiative amount to $21,612, an 83% reduction in our combined AMI and Docker image storage costs. Moreover, this figure doesn’t capture the full impact of the Cost Detective. By implementing this solution, we’ve also prevented all the compounding costs that would have accumulated if we had continued without a structured deletion process.

It’s worth noting that for our newer AMI build processes, we now rely on AWS EC2 Image Builder, which provides native AMI lifecycle management. However, this service wasn’t available when we initially set up our AMI creation workflows. The AMI & Docker Cost Detective continues to play a crucial role in managing our legacy systems that predate our EC2 Image Builder implementation.

This initiative underscores the importance of implementing structured lifecycle management for all types of infrastructure images. By addressing both AMIs and Docker images with a unified approach, we’ve significantly reduced our storage costs while maintaining the flexibility and reliability our development and deployment processes require.

ElasticIP Management Optimization: $9,482

Our cost-saving journey led us to an often-overlooked area: Elastic IP (EIP) management. We discovered we were incurring significant costs for unused public Elastic IPs, a situation stemming from our unique operational requirements.

At Grabyo, our encoders frequently need to connect to client infrastructure, necessitating IP allowlisting for content access. This requirement obliges us to maintain a pool of Elastic IPs for our client encoders. However, our investigation revealed inefficiencies in our EIP management, particularly in client offboarding processes.

To address this, we developed “ElasticIP Detective,” a system that maps all Elastic IPs across our accounts to their corresponding Grabyo organization or infrastructure. This tool alerts us when organizations leave, enabling prompt release of associated IPs and preventing unnecessary ongoing expenses.

We also identified optimization opportunities in our application server and database setups. Historically, these ran in public subnets, protected by security groups. As part of this initiative, we migrated these components to private subnets, eliminating the need for public IP allocation.

The combined effect of improved EIP management and infrastructure relocation was substantial, yielding annual savings of $9,482.

Conclusion of Part I

In this first part of our cost optimization journey, we’ve explored the quick wins and performance enhancements that led to significant savings. Through our efforts in tuning live encoders, optimizing Fargate performance, rethinking our database architecture, and implementing smart detection systems, we’ve achieved a total saving of $409,644 annually.

Our journey demonstrates the power of systematic analysis and targeted optimizations. We’ve seen how small changes, like switching from gp2 to gp3 volumes, can lead to substantial savings. We’ve also learned the importance of challenging established practices, as seen in our database consolidation efforts.

But our journey doesn’t end here. In Part II, we’ll delve into our strategic changes and long-term optimizations, including service migrations and retirement of unused systems. We’ll see how these moves not only further reduced our costs but also streamlined our operations and improved our overall system architecture. Stay tuned for more insights and lessons learned in our AWS cost optimization journey!