Anti-Pattern: Production Failure Scenario #

Every Terraform-caused infrastructure incident can be traced back to one of a few repeating anti-patterns: poorly managed state, lifecycles not configured for critical resources, rushed applies without plan review, or configurations never tested in staging. This article covers the most common failure scenarios — not to scare you, but so you can build the right safety net before these scenarios happen in your environment.

flowchart TD
    A["Common Failure\nPatterns"] --> B["State corruption"]
    A --> C["Accidental destroy"]
    A --> D["No staging test"]
    A --> E["Credential leak"]
    B --> F["Safety Net"]
    C --> F
    D --> F
    E --> F

    style A fill:#ef4444,stroke:#dc2626,color:#fff
    style B fill:#f59e0b,stroke:#d97706,color:#fff
    style C fill:#f59e0b,stroke:#d97706,color:#fff
    style D fill:#f59e0b,stroke:#d97706,color:#fff
    style E fill:#f59e0b,stroke:#d97706,color:#fff
    style F fill:#10b981,stroke:#059669,color:#fff

Scenario 1 — Accidental terraform destroy #

INCIDENT TIMELINE:

  09:00  An engineer intends to destroy the dev environment to save costs
  09:01  terraform workspace list  → forgot to check, still in the production workspace
  09:02  terraform destroy          → pressed "yes" without reading the output
  09:03  All production resources start getting deleted
  09:05  Monitoring alerts go off — production is down
  09:07  The team realizes what happened

  Impact: 2 hours of downtime, unbacked-up data lost

PREVENTION:

  1. Workspace safety in the shell prompt
     # .zshrc / .bashrc
     terraform_prompt() {
       if [ -f .terraform/environment ]; then
         workspace=$(cat .terraform/environment 2>/dev/null || echo "default")
         if [ "$workspace" = "production" ]; then
           echo " ⚠️ PRODUCTION"
         fi
       fi
     }
     PS1='$(terraform_prompt) $ '

  2. prevent_destroy for critical resources
     lifecycle {
       prevent_destroy = true
     }
     # Terraform errors before the destroy starts

  3. Extra confirmation for production
     # Script wrapper
     if [[ "$1" == "destroy" ]] && grep -q "production" .terraform/environment; then
       read -p "PRODUCTION DESTROY! Type the environment name to confirm: " confirm
       [ "$confirm" != "production" ] && exit 1
     fi
     terraform "$@"

  4. IAM policies that don't allow destroys in production
     # IAM policy denying DeleteDBInstance, TerminateInstances, etc.
     # for roles used by developers — only the CI/CD pipeline can do it

Scenario 2 — State Corruption #

INCIDENT TIMELINE:

  14:00  Engineer A runs terraform apply on their laptop
  14:00  Engineer B runs terraform apply in the CI/CD pipeline (simultaneously)
  14:01  State locking is not configured (local backend or S3 without DynamoDB)
  14:02  Both applies finish, the last saved state overwrites the other
  14:03  State no longer reflects the actual infrastructure
  14:04  terraform plan shows strange changes
  14:05  The team doesn't know what's right — state or cloud reality?

PREVENTION:

  1. Remote backend with locking is MANDATORY
     terraform {
       backend "s3" {
         bucket         = "terraform-state"
         key            = "production/terraform.tfstate"
         region         = "ap-southeast-1"
         dynamodb_table = "terraform-state-lock"  ← THIS IS REQUIRED
         encrypt        = true
       }
     }

  2. No applies from laptops for production
     - All production applies only from the CI/CD pipeline
     - IAM roles on laptops: read-only to production

  3. Automatic state backups
     # S3 bucket with versioning
     resource "aws_s3_bucket_versioning" "state" {
       bucket = aws_s3_bucket.terraform_state.id
       versioning_configuration {
         status = "Enabled"
       }
     }
     # If state is corrupted: restore the previous version from S3

Scenario 3 — Downtime from an Unexpected Replace #

INCIDENT TIMELINE:

  10:00  An engineer changes a tag on aws_db_instance in the Terraform configuration
  10:01  terraform plan — long output, the engineer scrolls quickly
  10:02  In the middle of the output (which wasn't read): -/+ aws_db_instance.main (forces replacement)
         It turns out the RDS minor version in the parameter group changed and forces a recreate
  10:03  terraform apply -auto-approve (in CI/CD)
  10:04  RDS is destroyed, then recreated → 15 minutes of downtime
  10:05  No data lost (there are backups), but there's a 15-minute outage

PREVENTION:

  1. Read the ENTIRE plan output before applying
     Don't skip the middle part — -/+ can be hidden anywhere

  2. Check the destroy/replace count in the summary
     Plan output always ends with a summary like:
     "Plan: 1 to add, 2 to change, 1 to destroy."
     Every number in "to destroy" needs attention

  3. Automated destroy checks in the pipeline
     terraform plan -out=tfplan
     DESTROY=$(terraform show -json tfplan | jq '[.resource_changes[]
       | select(.change.actions[] == "delete")] | length')
     if [ "$DESTROY" -gt 0 ]; then
       echo "ERROR: $DESTROY resources will be deleted!"
       exit 1  # Block the pipeline
     fi

  4. create_before_destroy for databases
     resource "aws_db_instance" "main" {
       lifecycle {
         create_before_destroy = true
       }
     }
     # There's still downtime for the connection switch, but it's shorter

Scenario 4 — Secrets Exposed via Plan Output #

INCIDENT TIMELINE:

  The CI/CD configuration posts the terraform plan output to the PR as a comment
  The plan output contains a value from a variable that was forgotten to be marked sensitive = true
  The PR is public (open source project)
  An API key is exposed to the internet for several hours before anyone notices

PREVENTION:

  1. Mark ALL sensitive variables and outputs
     variable "api_key" {
       type      = string
       sensitive = true   ← REQUIRED for sensitive values
     }

  2. Filter the plan output before posting to the PR
     terraform show -no-color tfplan \
       | grep -v "sensitive" \   # Filter lines containing "sensitive"
       > plan_filtered.txt

  3. Don't post plans to public PRs if there are sensitive resources
     Use a link to the pipeline log requiring authentication
     instead of posting directly in the comment

  4. Scan the plan output before posting
     # Check for patterns that look like credentials
     grep -E "AKIA|password|secret|api_key" plan_output.txt && exit 1

Scenario 5 — Reversed Dependencies Causing Failure #

INCIDENT TIMELINE:

  Team A manages the networking configuration
  Team B manages the application configuration — uses terraform_remote_state
  to the networking state to get subnet IDs

  Team A decides to rename the output "public_subnet_ids"
  to "subnet_ids_public" (naming refactor)

  Team A applies → succeeds (the new output exists)
  Team B plans → ERROR: output "public_subnet_ids" doesn't exist in remote state

  Team B must update their configuration before they can plan/apply

PREVENTION:

  1. Explicit, stable output contracts
     # Don't rename outputs already used by other configurations
     # If a rename is needed: add an alias first

     # networking outputs.tf:
     output "public_subnet_ids" {
       value       = aws_subnet.public[*].id
       description = "DEPRECATED: Use subnet_ids_public"
     }
     output "subnet_ids_public" {
       value = aws_subnet.public[*].id
     }

  2. Coordinate between teams before breaking output changes
     Notify all teams using the remote state
     Give a migration window before the old output is removed

  3. Consider input variables instead of remote state
     Looser coupling — team B receives values via variables
     rather than directly from team A's state
     Naming changes in team A don't directly break team B

Production Safety Net Checklist #

BEFORE EVERY APPLY TO PRODUCTION:

PLAN REVIEW:
  □ Read the entire plan output, don't skip
  □ No unintended -/+ (replace) operations
  □ The "to destroy" count matches expectations (ideally 0)
  □ The changed resources match the configuration changes made

STATE & LOCKING:
  □ Remote backend with locking active
  □ No other plan is currently running (check the DynamoDB lock table)
  □ State is backed up (S3 versioning active)

CREDENTIALS & ACCESS:
  □ Using the correct workspace/profile (not the dev profile in production)
  □ The right IAM role (apply role, not a plan-only role)

POST-APPLY:
  □ Verify resources are running after the apply
  □ Run smoke tests or health checks
  □ Notify the team that the apply succeeded

IF THERE'S A PROBLEM:
  □ Don't panic-apply again — identify the problem first
  □ Check the state: terraform state list and terraform state show
  □ If rollback is needed: revert the commit and apply the previous version
  □ Document the incident for a post-mortem

flowchart LR
    A["Plan\nreview"] --> B["Staging\ntest"]
    B --> C["Backup\nstate"]
    C --> D["Apply\nproduction"]
    D --> E["Verify\noutput"]

    style A fill:#3b82f6,stroke:#1e40af,color:#fff
    style B fill:#3b82f6,stroke:#1e40af,color:#fff
    style C fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style D fill:#10b981,stroke:#059669,color:#fff
    style E fill:#10b981,stroke:#059669,color:#fff

Post-Incident Review Template #

## Incident Report: [Date]

### Summary
- **Duration**: X minutes/hours
- **Impact**: [affected resources/services]
- **Root Cause**: [main cause]

### Timeline
- HH:MM - Terraform apply started
- HH:MM - Error detected
- HH:MM - Rollback started
- HH:MM - Service recovered

### Lessons Learned
- [ ] Add a test for [scenario]
- [ ] Update the policy for [check]
- [ ] Improve monitoring for [metric]

Monitoring Terraform Changes #

# After an apply, verify resources are working

# Health check for EC2
aws ec2 describe-instance-status --instance-ids i-12345

# Health check for RDS
aws rds describe-db-instances --db-instance-identifier prod-db \
  --query 'DBInstances[0].DBInstanceStatus'

# Health check for ALB
aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:...

# Automated verification in CI/CD
terraform output -json | jq -r '.health_check_url.value' | xargs curl -f

Summary #

  • Accidental terraform destroy is prevented with prevent_destroy, showing the workspace in the shell prompt, and a “no production destroys from laptops” policy.
  • State corruption happens when two applies run simultaneously without locking — a remote backend with DynamoDB locking is an absolute requirement, not optional.
  • Unexpected replaces are caused by changes that “force” resource recreates — an automated check on the destroy count in the plan output can block the pipeline before applying.
  • Secrets in plan output happen from forgetting sensitive = true — filter the output before posting to public PRs and scan for credential-like patterns.
  • Reversed cross-configuration dependencies can be mitigated with stable output contracts, deprecation periods before renames, and cross-team coordination.
  • The pre-production apply checklist is a small investment preventing large incidents — read the entire plan, verify the workspace, and always have post-apply verification.
← Previous: Anti-Pattern: Over-Complex Module
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact