Abstract

Power Digital Marketing, a rapidly growing digital marketing agency, faced scalability challenges with their existing Terraform Cloud infrastructure. With a complex setup supporting numerous clients and a growing number of resources, they needed a far more stable, cost-effective solution. Masterpoint stepped in, leveraging their deep Terraform expertise to migrate Power Digital from Terraform Cloud to Spacelift, a more flexible and cost-effective platform. Through careful planning and execution, Masterpoint successfully migrated over 43,000 resources, optimized the infrastructure, and implemented automation to support Power Digital Marketing’s future growth.

The result was a 10x reduction in infrastructure costs, significantly faster plan and apply cycles, and a more resilient, scalable infrastructure that can easily onboard new clients

tldr;

• Over 10x reduction in projected infrastructure costs.
• Faster plan and apply cycles: 3 minutes, sub 1 minute in some cases.
• Nearly eliminated failed/aborted plan and apply runs.
• More resilient and scalable infrastructure.
• Easier client onboarding.
• Enhanced disaster recovery.

Are you facing similar challenges?
Contact Masterpoint today, we’ll evaluate your infrastructure and develop a comprehensive plan and solution.

Article

Power Digital Marketing is a modern digital marketing powerhouse, working with the world’s most recognizable brands, including Stripe, Figma, and Casper. As a technology-driven company, Power Digital’s success requires delivering best-in-class marketing services built on a robust and flexible platform infrastructure.

The company’s rapid growth, while a testament to its success, brought with it a set of challenges that impeded further expansion. As Power Digital onboarded more clients and expanded its service offerings, the existing infrastructure limitations became increasingly apparent. The company found itself at a crossroads, needing to address these challenges to maintain its competitive edge while it continued to scale.

The Challenge

Monolithic Terraform Configuration (Terralith)

At the heart of Power Digital Marketing’s infrastructure challenges lay a monolithic Terraform configuration: a “Terralith.” This massive configuration had evolved over time to encompass all the infrastructure code for every client deployment within a single, unwieldy terraform state file.

The Terralith presented several significant challenges:

1. Complexity: As the configuration grew, it became increasingly difficult for developers to navigate the codebase; the root module had grown to over 500 configuration files, and was growing each day as new clients were added.
2. Maintenance Overhead: Updates and modifications to the infrastructure required careful consideration of potential impacts across the entire system.
3. High Blast Radius: Even small changes could not be isolated and could have far-reaching consequences, potentially affecting all clients or services.
4. Long Feedback Cycles: The size and complexity of the Terralith led to extended periods between making a change and seeing its effects, slowing down development, troubleshooting, and delivery processes.

This monolithic structure, while initially manageable, had become a significant bottleneck in Power Digital’s ability to scale and adapt to business needs.

Scalability Issues with Growing Resources and Client Base

The company’s platform team found itself managing over 43,000 resources…

a scale that pushed their existing Terraform Cloud setup to continually breaking.

The scalability issues manifested in several ways:

1. Frequent Timeouts: As the number of resources grew, Terraform Cloud struggled to process plan/apply loops within allocated time limits, resulting in frequent timeouts.
2. High Failure Rate: Over 50% of Terraform runs ended in failure, creating a frustrating and inefficient workflow for the development team.
3. Disrupted Automation: The platform team could no longer rely on Terraform Cloud to reliably deploy new infrastructure; they often had to resort to manual operations to provision resources for new clients.
4. Extended Plan and Apply Cycles: What should have been quick operations turned into lengthy processes, with plan and apply cycles taking up to 25 minutes to complete.

These scalability issues significantly impacted the efficiency of development and deployment processes, causing delays, frustration among the team, and affecting the quality of service delivered to clients.

High Infrastructure Costs and Financial Burden

Power Digital’s infrastructure challenges were not limited to technical issues. The company was grandfathered into an outdated Terraform Cloud pricing model; scaling further meant adopting a new pricing plan that was exponentially more expensive without guarantees of improved performance.

This fi nancial pressure added another layer of urgency to Power Digital’s need for a more scalable and cost-eff ective infrastructure solution.

Delayed Client Onboarding and Reduced Effi ciency

The cumulative eff ect of these challenges had a direct impact on Power Digital’s core business operations: onboarding new clients onto the platform was the primary revenue driver, and the existing platform was a bottleneck.

Key issues included:

1. Slow Client Onboarding: The complex Terraform Cloud setup and frequent slowdowns negatively impacted the client onboarding process, causing delays and aff ecting client satisfaction.
2. Resource Misallocation: Developers and operations staff found themselves spending an inordinate amount of time managing infrastructure issues instead of focusing on delivering value and new features
3. Lost Productivity: Power Digital estimated that at least 63 hours of engineer time per month were spent just waiting on plan/apply loops and debugging broken Terraform runs.

While these effi ciency issues were felt by engineers on the platform team immediately, the long-term eff ect goes right to the bottom line: slow, broken client onboarding directly impacts revenue and business growth potential.

Limited Ability to Scale and Support Long-term Goals

As Power Digital Marketing continued to grow its client base, it became increasingly clear that the current infrastructure would not be able to support their long-term goals and objectives. The existing Terraform Cloud setup was not designed to handle the scale at which the company was now operating.

These types of limitations posed a signifi cant threat to Power Digital’s long-term success and market position.

The Solution

To address these challenges, Power Digital Marketing needed a partner with deep expertise in Terraform and infrastructure management. Masterpoint developed a full-featured solution to migrate Power Digital from Terraform Cloud to a more flexible and cost-effective platform, as well as crafting a customized, modular IaC configuration that would handle their current and future scaling needs.

Assessment and Planning

Masterpoint’s approach began with a thorough assessment of Power Digital’s existing infrastructure:

1. Full IAC Audit: Conducted a comprehensive Infrastructure-as-Code (IAC) audit of Power Digital Marketing’s existing Terralith and Terraform Cloud deployment.
2. Stakeholder Interviews: Engaged with various teams at Power Digital to understand their specific needs and pain points.
3. Challenge Identification: Pinpointed key challenges and limitations, including scalability issues, slow performance, and high costs.
4. Future-State Visioning: Worked with Power Digital leadership to define the desired future state of their infrastructure.

Based on this assessment, Masterpoint developed a detailed migration strategy that accounted for several critical factors:

• Minimizing downtime during the migration process
• Ensuring data integrity throughout the transition
• Maintaining, and enhancing security standards

One of the most critical steps in any platform or infrastructure migration is having a detailed, well-formed plan that considers the unique challenges and needs of a given organization and its infrastructure configuration. Masterpoint’s IaC Audit delivers a clear framework that helps stakeholders feel like partners in their infrastructure, rather than just another client.

Platform Migration: Terraform Cloud to Spacelift

After considering several implementation options, Masterpoint recommended migrating to Spacelift, a leading Terraform Automation and Collaboration Software (TACOS). This decision was based on several key advantages that Spacelift offered over Terraform Cloud:

1. Cost-Effective Pricing Model: Spacelift’s pricing structure was much more in line with Power Digital’s usage patterns and growth trajectory, offering significant cost savings.
2. Superior Resource Management: Spacelift provides more granular control over resource management and deployment.
3. Advanced Automation Features: Spacelift offers an impressive amount of customization through a deep integration with Open Policy Agent and 3rd-party integrations.
4. Enhanced Operational Guardrails: Spacelift enables users to define and enforce control policies directly within the platform, making it easy to develop granular operational and security controls without needing 3rd-party tools.
5. Scalability: Spacelift makes it easy to handle large numbers of resources and environments; users can easily decompose large configurations into manageable modules that are easily connected with automation.

With Spacelift, Masterpoint was able to migrate over 43,000 resources from Terraform Cloud without issue.

Decomposing the Terralith

Power Digital had an unwieldy Terralith containing over 43,000 Terraform resources which contributed to all the performance and maintainability issues detailed above; a complete refactor with an emphasis on modularity was a must. However, moving to a new platform doesn’t mean the business can be put on pause. The Masterpoint team needed to craft a strategy that enabled the new system to be built without interrupting ongoing work.

Masterpoint’s approach to decomposing the Terralith included:

1. Modularization: Breaking down the monolithic Terralith into individual client deployments with isolated resources and state.
2. Migration Strategy: A new, modularized system was developed and run in parallel with the existing Terralith. This allowed for new clients to be onboarded using the new system without impacting existing clients or disrupting ongoing business operations. Once the new system was fully deployed for new clients, the team proceeded to decompose the original Terralith and migrate existing clients to the new modularized environments.
3. Standardization: Developing standardized modules for common infrastructure components, promoting best practices around module usage with tofu Workspaces and removing repetitive code anti-patterns.
4. Dependency Management: Clearly defining and managing dependencies between modules to ensure smooth operations.
5. Version Control Optimization: Restructuring the repository to better support the new modular approach, improving collaboration and code management for the platform engineering team. This restructuring also facilitated a smoother transition for existing clients and easier onboarding for new clients during the migration process.

This modularization, combined with Spacelift’s automation capabilities, enabled sub-3-minute init/plan/ apply deployments for new clients, a dramatic improvement over a configuration that took 25 minutes just to complete a plan.

Migration to OpenTofu

One of the most impressive aspects of the migration, and one that showcased the capabilities of the OpenTofu community, Masterpoint simultaneously migrated over 43,000 resources from Terraform to the open-sourcefriendly OpenTofu project. With the current uncertainty around Terraform and its licensing structure, moving to OpenTofu offers several strategic advantages:

1. Licensing Flexibility: Moving to OpenTofu eliminated risks due to potential future licensing changes in Terraform and legal issues.
2. Community Support: The growing OpenTofu community provides community support and tooling reminiscent of the early Terraform community.
3. Cost Savings: By eliminating their dependence on commercial Terraform, Power Digital protected themselves from vendor lock-in and cost increases down the road.

Masterpoint is a proud OpenTofu community member; we believe the project is the future of IaC technology, and migrations like this one highlight why.

Knowledge Transfer and Training

Masterpoint wants customers to be partners in their IaC journey, which means delivering turn-key IaC solutions that includes training the current engineering team on how to manage and maintain them. Masterpoint provided a comprehensive knowledge transfer and training to Power Digital Marketing’s team.

The training program included:

1. Hands-on Workshops: Practical sessions on using Spacelift and OpenTofu effectively.
2. Best Practices Sessions: Guidance on infrastructure-as-code best practices and patterns.
3. Troubleshooting Guides: Detailed documentation on common issues and how to resolve them.
4. Continuous Learning Plans: Establishing a framework for ongoing learning and skill development.

This comprehensive training ensured that Power Digital’s team was well-equipped to not only manage and maintain the new infrastructure platform, but to iterate and deliver their own improvements and features long-term.

The Results

The migration from Terraform Cloud to Spacelift and OpenTofu, led by Masterpoint, yielded impressive results for Power Digital Marketing:

1. Cost Reduction: There was a 10x reduction in projected infrastructure automation costs compared to the required Terraform Cloud upgrade and subsequent cost-increase. By adopting Spacelift, Power Digital is getting vastly superior infrastructure automation capabilities that cost less than $500/month.
2. Improved Performance: Plan and apply cycles for creating client infrastructure reduced from 25 minutes to sub-3-minute completion times. There is an additional spin-up phase to create Spacelift resources that takes single digit minutes, but we are not including that in our calculation here.
3. Improved Reliability: We’ve completely eliminated the failed plan and apply runs that were resulting from Terraform running out of memory. This has greatly improved overall system stability and eliminated the platform team’s operational toil.
4. Increased Scalability: The new infrastructure means the platform is no longer a bottleneck for onboarding new clients, enabling the business to grow and scale quickly. As a benefit of Masterpoint’s work, Power Digital has been able to onboard a total of 100+ new clients without issue in the past 60 days since migration.
5. Enhanced Security and Compliance: Improved access control, better audit trails, and standardized security practices across client deployments.
6. Better Disaster Recovery: Outages require a nimble response to prevent client impact: faster cycles, better automation reliability, and infrastructure isolation means the platform team can quickly stand up new infrastructure in response to trouble.

Takeaways

Power Digital Marketing’s move from Terraform Cloud to Spacelift and OpenTofu shows the importance of taking time to evaluate your infrastructure and asking questions:

1. Is our infrastructure automation cost-eff ective?
2. Does it meet our current needs?
3. Can it meet our future demands?

It’s ok if the answer is “no” to any of these questions, especially the last one. This project you just read about was not a result of a failure in design of the original system, but rather an evolution to remove scaling limits – a good problem to have.

By leveraging Masterpoint’s expertise, Power Digital has overcome significant challenges, reduced costs, and positioned themselves for continued growth and success.

If your organization is experiencing similar challenges with infrastructure scalability, cost management, or effi ciency, it may be time to consider a change. Whether you just need an external perspective on your current setup or want someone to take IaC off your hands completely, we can help. Get in touch with us to schedule an assessment to see how we can improve your infrastructure and save you money and time.