Scale Terraform #

A Terraform configuration that starts small — one main.tf, one terraform.tfstate — can grow to manage hundreds of resources in a single configuration. At that point, problems start to appear: terraform plan takes a long time because it must refresh all resources, one mistake can affect unrelated infrastructure, and the blast radius of every apply becomes too large. Scaling Terraform isn’t about adding more machine specs to the pipeline runner, but about splitting the configuration based on the right principles.

flowchart TD
    A["Small\n1 state file"] --> B["Medium\nfew states"]
    B --> C["Large\nmany states"]
    C --> D["Problems:\nSlow plan\nBig blast radius"]
    D --> E["Split by\nenvironment"]
    D --> F["Split by\ncomponent"]
    D --> G["Split by\nteam"]

    style A fill:#10b981,stroke:#059669,color:#fff
    style B fill:#3b82f6,stroke:#1e40af,color:#fff
    style C fill:#f59e0b,stroke:#d97706,color:#fff
    style D fill:#ef4444,stroke:#dc2626,color:#fff
    style E fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style F fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style G fill:#8b5cf6,stroke:#6d28d9,color:#fff

Signs That a Configuration Needs Splitting #

SIGNALS THAT A CONFIGURATION IS TOO BIG:

  ✗ terraform plan takes more than 5 minutes (too many API refreshes)
  ✗ One apply affects dozens of unrelated resources
  ✗ Different teams frequently conflict over editing the same files
  ✗ There are "don't touch" resources mixed with routinely changing ones
  ✗ Testing a small change requires waiting for a plan of the entire infrastructure
  ✗ The state file is more than a few MB (thousands of resources)
  ✗ The team no longer knows what resources are in this configuration

Split Principles: Lifecycle and Blast Radius #

The most effective way to split configurations isn’t by resource type, but by two questions: how often does it change, and what’s the impact if something goes wrong?

LIFECYCLE PRINCIPLE:
  Resources that change at different frequencies should be separated.

  Changes rarely (months/years):
    VPC, subnets, routing, peering
    IAM roles, SCPs, organizations
    DNS zones (not records)
    → One configuration, applies very rarely

  Changes occasionally (weeks/months):
    Kubernetes clusters, RDS instances
    Load balancers, security groups
    S3 buckets, KMS keys
    → A separate configuration per domain

  Changes often (days/weeks):
    Kubernetes deployments
    ECS task definitions
    Lambda functions
    DNS records
    → A separate configuration, can auto-apply

BLAST RADIUS PRINCIPLE:
  Critical resources (expensive downtime) are separated from non-critical ones.
  If there's a bug applying the database configuration,
  it must not affect networking.

Layering Patterns: Dependencies Between Configurations #

LAYERED ARCHITECTURE:

  Layer 1: Foundation (changes most rarely)
    ├── VPC, subnets, internet gateway
    ├── Route53 zones
    ├── Basic IAM roles
    └── S3 bucket for state
    Outputs: vpc_id, subnet_ids, route53_zone_id

  Layer 2: Platform (changes during infrastructure upgrades)
    ├── EKS cluster / ECS cluster
    ├── RDS instances
    ├── ElastiCache
    └── Load balancers
    Inputs: vpc_id, subnet_ids (from layer 1)
    Outputs: cluster_endpoint, db_endpoint, lb_dns_name

  Layer 3: Application (changes most often)
    ├── Kubernetes deployments
    ├── ECS task definitions
    ├── DNS records (pointing at the LB from layer 2)
    └── Lambda functions
    Inputs: cluster_endpoint, db_endpoint (from layer 2)

  Each layer only depends on the layer below it.
  Changes in an upper layer don't affect lower layers.

# Layer 3 reads outputs from Layer 2 via remote state
data "terraform_remote_state" "platform" {
  backend = "s3"
  config = {
    bucket = "my-terraform-state"
    key    = "platform/terraform.tfstate"
    region = "ap-southeast-1"
  }
}

resource "aws_route53_record" "app" {
  zone_id = data.terraform_remote_state.platform.outputs.route53_zone_id
  name    = "api.example.com"
  type    = "CNAME"
  ttl     = 300
  records = [data.terraform_remote_state.platform.outputs.lb_dns_name]
}

Directory Structure for Large Infrastructure #

infra-repo/
  ├── modules/                      ← Shared, versioned modules
  │   ├── networking/
  │   ├── eks-cluster/
  │   └── rds/
  │
  ├── foundation/                   ← Layer 1
  │   ├── networking/
  │   │   ├── main.tf
  │   │   ├── outputs.tf
  │   │   └── backend.tf
  │   └── iam/
  │
  ├── platform/                     ← Layer 2
  │   ├── eks/
  │   ├── databases/
  │   └── load-balancers/
  │
  ├── applications/                 ← Layer 3
  │   ├── api-service/
  │   ├── worker-service/
  │   └── frontend/
  │
  └── scripts/
      ├── plan-all.sh               ← Plan all layers in the right order
      └── apply-all.sh

# plan-all.sh — plan all layers in the correct order
#!/bin/bash
set -e

ENVIRONMENT=${1:-staging}

echo "=== Layer 1: Foundation ==="
cd foundation/networking
terraform plan -var="environment=$ENVIRONMENT"

echo "=== Layer 2: Platform ==="
cd ../../platform/eks
terraform plan -var="environment=$ENVIRONMENT"

echo "=== Layer 3: Applications ==="
cd ../../applications/api-service
terraform plan -var="environment=$ENVIRONMENT"

Parallel vs Serial Execution #

SERIAL (the default for dependencies):
  Layer 1 finishes → Layer 2 starts → Layer 3 starts
  Suitable for tight dependencies between layers

PARALLEL (for independent configurations):
  Layer 2: eks, databases, load-balancers can be applied simultaneously
  Each only depends on Layer 1 (already finished)

  In a CI/CD matrix:
  jobs:
    apply-platform:
      strategy:
        matrix:
          component: [eks, databases, load-balancers]
        max-parallel: 3
      steps:
        - run: terraform apply platform/${{ matrix.component }}/tfplan

  CAUTION: Parallel is only safe if there are no dependencies between
  the components being run in parallel. If EKS needs outputs from
  databases, they must run serially.

Managing Many Environments with One Configuration #

# Pattern: One configuration, three environments, different backends

# environments/dev/backend.tf
terraform {
  backend "s3" {
    bucket = "terraform-state-dev"
    key    = "platform/eks/terraform.tfstate"
    region = "ap-southeast-1"
  }
}

# environments/production/backend.tf
terraform {
  backend "s3" {
    bucket = "terraform-state-production"
    key    = "platform/eks/terraform.tfstate"
    region = "ap-southeast-1"
  }
}

# The main configuration (symlinked or shared) uses variables
variable "environment" {
  type = string
}

module "eks" {
  source = "../../modules/eks"

  cluster_name    = "myapp-${var.environment}"
  node_count      = var.environment == "production" ? 5 : 2
  instance_type   = var.environment == "production" ? "m5.xlarge" : "t3.medium"
}

flowchart LR
    A["1 big state\n(200 resources)"] -->|"split"| B["networking\nstate"]
    A -->|"split"| C["compute\nstate"]
    A -->|"split"| D["database\nstate"]
    B --> E["Fast plan\nSmall blast radius"]
    C --> E
    D --> E

    style A fill:#ef4444,stroke:#dc2626,color:#fff
    style B fill:#3b82f6,stroke:#1e40af,color:#fff
    style C fill:#3b82f6,stroke:#1e40af,color:#fff
    style D fill:#3b82f6,stroke:#1e40af,color:#fff
    style E fill:#10b981,stroke:#059669,color:#fff

State Segmentation Strategy #

As infrastructure grows, one state file becomes a bottleneck.

# RECOMMENDATION: Split state by blast radius

# Per environment + per component:
infrastructure/
├── networking/
│   ├── dev/
│   ├── staging/
│   └── production/
├── compute/
│   ├── dev/
│   ├── staging/
│   └── production/
└── database/
    ├── dev/
    ├── staging/
    └── production/

flowchart TD
    A["Monolithic\nState"] -->|"Split"| B["networking/\nstate"]
    A -->|"Split"| C["compute/\nstate"]
    A -->|"Split"| D["database/\nstate"]
    A -->|"Split"| E["security/\nstate"]

    style A fill:#ffebee,stroke:#c62828
    style B fill:#e3f2fd,stroke:#1565c0
    style C fill:#e8f5e9,stroke:#2e7d32
    style D fill:#fff3e0,stroke:#e65100
    style E fill:#f3e5f5,stroke:#6a1b9a

Terragrunt for Scale #

# Terragrunt helps manage many modules at large scale
# Features: DRY config, dependency management, parallel execution

# Install
brew install terragrunt

# Terragrunt structure
infrastructure/
├── terragrunt.hcl           # Root config
├── networking/
│   ├── terragrunt.hcl       # Module config
│   └── main.tf
├── compute/
│   ├── terragrunt.hcl
│   └── main.tf
└── database/
    ├── terragrunt.hcl
    └── main.tf

# Apply everything at once
terragrunt run-all apply

# Apply with dependency ordering
terragrunt run-all apply --terragrunt-parallelism 3

Module Layering Strategy #

LAYERING STRATEGY:

Layer 1: Foundation
├── VPC, Subnets, Security Groups
├── IAM roles, KMS keys
└── Terraform state backend

Layer 2: Platform
├── EKS/GKE/ECS cluster
├── RDS instances
├── ElastiCache
└── Load balancers

Layer 3: Application
├── Application deployments
├── DNS records
├── CDN configuration
└── Monitoring dashboards

DEPENDENCY: Layer 3 → Layer 2 → Layer 1

flowchart TD
    subgraph L1["Layer 1: Foundation"]
        VPC["VPC"]
        IAM["IAM"]
        KMS["KMS"]
    end
    
    subgraph L2["Layer 2: Platform"]
        EKS["EKS"]
        RDS["RDS"]
        ALB["ALB"]
    end
    
    subgraph L3["Layer 3: Application"]
        APP["App Deploy"]
        DNS["DNS"]
        MON["Monitoring"]
    end
    
    L1 --> L2
    L2 --> L3
    
    style L1 fill:#e3f2fd,stroke:#1565c0
    style L2 fill:#fff3e0,stroke:#e65100
    style L3 fill:#e8f5e9,stroke:#2e7d32

Large State Management #

# State > 100MB can cause slow plans/applies
# SOLUTION: Split into multiple states

# Indicators that state is too large:
# - terraform plan > 5 minutes
# - High memory usage
# - Frequent lock contention
# - Many unrelated changes in one apply

# Split strategy by service:
# networking/terraform.tfstate
# compute/terraform.tfstate
# database/terraform.tfstate
# monitoring/terraform.tfstate

# Each state = one responsible team

# State file size monitoring
ls -lh terraform.tfstate

# State summary
terraform state list | wc -l

# Check the state file contents
terraform show -json | jq '.values.root_module.resources | length'

Summary #

• Signs a configuration needs splitting: plans taking more than 5 minutes, teams frequently conflicting over the same files, “don’t touch” resources mixed with routinely changing ones.
• Split by lifecycle and blast radius, not by resource type — rarely changing and critical resources are separated from frequently changing and less critical ones.
• The layering pattern (foundation → platform → application) gives clear dependencies: each layer only depends on the layer below it, upper changes don’t affect lower layers.
• Use terraform_remote_state to read outputs between separate configurations — but remember this creates coupling between configurations.
• Components in the same layer (EKS, database, load balancer) can be applied in parallel if there are no dependencies between them.
• Start simple — don’t prematurely decompose a configuration before there are real problems. Split only when the signals clearly appear.

← Previous: Lifecycle Management   Next: Performance Optimization →