Drift Detection #

Drift is the condition where the actual infrastructure running in the cloud differs from what’s defined in the Terraform configuration. It happens more often than you’d think — someone edits a security group directly in the console for debugging, an auto-scaling event changes the instance count, or a cloud service automatically modifies certain attributes. Undetected and unmanaged drift can become a source of hard-to-trace bugs and an unnoticed security hole.

What Is Drift #

Drift occurs when there’s a mismatch between three things: the Terraform configuration you wrote, the state Terraform stores, and the actual condition in the cloud.

flowchart TD
    subgraph "Ideal Condition (In Sync)"
        A1["Configuration (.tf)\nport 443 only"] -.->|"same"| B1["State (.tfstate)\nport 443 only"]
        B1 -.->|"same"| C1["Reality (cloud)\nport 443 only"]
    end

    subgraph "Drift Condition"
        A2["Configuration (.tf)\nport 443 only"] -.->|"same"| B2["State (.tfstate)\nport 443 only"]
        B2 -.->|"DIFFERENT ❌"| C2["Reality (cloud)\nport 443 + 22 + 8080"]
    end

    D["Someone opens ports 22\nand 8080 in the console"] --> C2

    style A1 fill:#e8f5e9,stroke:#2e7d32
    style B1 fill:#e8f5e9,stroke:#2e7d32
    style C1 fill:#e8f5e9,stroke:#2e7d32
    style A2 fill:#e3f2fd,stroke:#1565c0
    style B2 fill:#e3f2fd,stroke:#1565c0
    style C2 fill:#ffebee,stroke:#c62828
    style D fill:#fff3e0,stroke:#e65100

There are two types of drift to distinguish:

flowchart TD
    A["Types of Drift"] --> B["State ↔ Reality drift\n(Most common)"]
    A --> C["Configuration ↔ State drift\n(Rare, but dangerous)"]

    B --> B1["Cloud condition changed\nwithout Terraform knowing\nState still holds old data"]
    C --> C1["State edited manually\nor corrupted\n.tf configuration out of sync"]

    B1 --> D["Solution: terraform plan\nor terraform apply -refresh-only"]
    C1 --> E["Solution: terraform import\nor fix the state"]

    style B fill:#fff3e0,stroke:#e65100
    style C fill:#ffebee,stroke:#c62828
    style D fill:#e8f5e9,stroke:#2e7d32
    style E fill:#e8f5e9,stroke:#2e7d32

Detecting Drift #

There are several ways to detect drift, from the simplest to the most flexible.

flowchart TD
    A["Ways to Detect Drift"] --> B["terraform plan\n(refresh + diff)"]
    A --> C["terraform plan\n-refresh-only"]
    A --> D["terraform refresh\n(update state only)"]

    B --> B1["Refresh state from the cloud\nCalculate differences\nwith the configuration\nShow the change plan"]
    C --> C1["Refresh state from the cloud\nShow differences between\nstate and reality\nWITHOUT calculating the config diff"]
    D --> D1["Update state from the cloud\nNo output\nUseful for syncing state"]

    B --> B2["Best for:\nBefore routine applies"]
    C --> C2["Best for:\nScheduled drift detection"]
    D --> D2["Best for:\nSyncing state before\nother steps"]

    style B fill:#e3f2fd,stroke:#1565c0
    style C fill:#e8f5e9,stroke:#2e7d32
    style D fill:#fff3e0,stroke:#e65100

# Method 1: terraform plan (refreshes before planning)
terraform plan
# If there's drift, the plan will show changes
# even though you didn't change any configuration

# Output if there's drift:
# ~ aws_security_group.web will be updated in-place
#   ~ ingress = [
#       + {
#           + from_port   = 22
#           + protocol    = "tcp"
#           + to_port     = 22
#           + cidr_blocks = ["0.0.0.0/0"]
#         },
#     ]
#
# Note: Objects have changed outside of Terraform

# Method 2: terraform plan -refresh-only
# Only shows the differences between state and reality
# without calculating changes from the configuration
terraform plan -refresh-only

# Method 3: terraform refresh (update state only)
terraform refresh
# Updates the state file with the actual cloud condition
# without changing infrastructure and without detailed output

The Most Common Causes of Drift #

flowchart TD
    A["Causes of Drift"] --> B["Manual Changes\nin the Console"]
    A --> C["Automatic Cloud\nServices"]
    A --> D["Failed Partial\nApplies"]
    A --> E["Other Tools\n(Ansible, CLI)"]
    A --> F["Out-of-Sync\nState"]

    B --> B1["Developer opens SSH port\non a security group\nfor debugging\nForgot to revert"]
    C --> C1["Auto Scaling changes\ndesired capacity\nAWS rotates certificates\nManaged services apply minor updates"]
    D --> D1["Apply fails midway\nSome resources created\nsuccessfully, others not\nState is inconsistent"]
    E --> E1["kubectl apply changes\nthe same resource\nmanaged by Terraform\nAnsible configures servers"]
    F --> F1["Old or corrupted\nstate file\nDoesn't reflect the\nactual condition"]

    style B fill:#ffebee,stroke:#c62828
    style C fill:#fff3e0,stroke:#e65100
    style D fill:#fff3e0,stroke:#e65100
    style E fill:#fff3e0,stroke:#e65100
    style F fill:#fff3e0,stroke:#e65100

Responding to Drift #

When drift is detected, there are two strategic options. The choice depends on the context: whether the console change was intentional or not.

flowchart TD
    A["Drift Detected!"] --> B{"Was the console change\nintentional?"}

    B -->|"No (a mistake)"| C["Option 1: Terraform Wins\nRevert to the configuration"]
    B -->|"Yes (emergency fix)"| D{"Already documented\nin .tf?"}

    D -->|"Not yet"| E["Option 2: Reality Wins\nUpdate the .tf config"]
    D -->|"Already"| F["No problem\nThe plan will be empty"]

    C --> C1["terraform apply\nTerraform reverts\nthe infrastructure\nto the .tf configuration"]
    E --> E1["terraform apply -refresh-only\nUpdate the state\nfrom the actual condition\nThen edit .tf"]
    E1 --> E2["Commit the updated .tf\nto Git"]

    style C fill:#e3f2fd,stroke:#1565c0
    style E fill:#fff3e0,stroke:#e65100
    style F fill:#e8f5e9,stroke:#2e7d32

Option 1: Terraform Wins (Revert to the Configuration) #

# Restore the infrastructure to the configured condition
terraform apply
# Terraform will change the infrastructure to match .tf
# Useful when the manual change was a mistake

Option 2: Reality Wins (Accept the Change) #

# Update state to reflect the actual condition
terraform apply -refresh-only
# State is updated, the .tf configuration doesn't change

# Next step: update the .tf configuration to match
# the actual condition, then commit to version control

# Example: Drift detected — port 22 opened outside Terraform
# After team discussion, it's decided port 22 is indeed needed
# but must be managed by Terraform

# Update the .tf configuration:
resource "aws_security_group" "web" {
  name = "web-sg"

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # Add the rule that was previously opened manually:
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.office_cidr]  # Restrict to office IPs, not 0.0.0.0/0
  }
}
# Commit this configuration, then apply

Preventing Drift #

flowchart TD
    A["Drift Prevention Strategies"] --> B["1. Team Policy"]
    A --> C["2. Automated Detection"]
    A --> D["3. ignore_changes"]
    A --> E["4. Emergency Procedures"]

    B --> B1["All infrastructure changes\nmust go through Terraform\nIAM policy restrictions\nCode review for .tf"]
    C --> C1["Schedule plan -refresh-only\nevery working day\nAlert to Slack/email\nExit code 2 = drift"]
    D --> D1["For attributes that\nchange automatically\n(desired_capacity, tags,\ncertificate rotation)"]
    E --> E1["Manual edits allowed\nduring emergencies\nBUT must create a\nTerraform PR within 24 hours"]

    style B fill:#e3f2fd,stroke:#1565c0
    style C fill:#e8f5e9,stroke:#2e7d32
    style D fill:#fff3e0,stroke:#e65100
    style E fill:#f3e5f5,stroke:#7b1fa2

ignore_changes for Automatically Changing Attributes #

# Example: Auto Scaling Group — desired_capacity changes via scaling policies
resource "aws_autoscaling_group" "web" {
  name             = "web-asg"
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # desired_capacity can change via auto scaling policies
    # or manual scaling — ignore these changes in Terraform
    ignore_changes = [desired_capacity]
  }
}

Automated Drift Detection in CI/CD #

# Example: GitHub Actions for scheduled drift detection
# .github/workflows/drift-detection.yml

name: Terraform Drift Detection

on:
  schedule:
    - cron: '0 8 * * 1-5'  # Every working day at 8 AM

jobs:
  drift-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3

      - name: Terraform Init
        run: terraform init

      - name: Check for Drift
        id: plan
        run: |
          terraform plan -refresh-only -detailed-exitcode
          # Exit code 0: no changes
          # Exit code 1: error
          # Exit code 2: changes detected (drift found)          
        continue-on-error: true

      - name: Alert if there's drift
        if: steps.plan.outputs.exitcode == '2'
        run: |
          echo "Drift detected! Check Terraform plan output."
          # Send a notification to Slack, PagerDuty, etc.          

flowchart LR
    A["Schedule:\nEvery day\nat 8 AM"] --> B["terraform init"]
    B --> C["terraform plan\n-refresh-only\n-detailed-exitcode"]
    C --> D{"Exit code?"}

    D -->|"0: No changes"| E["✅ No drift\nLog and done"]
    D -->|"2: Changes detected"| F["⚠️ Drift!\nSend an alert"]
    D -->|"1: Error"| G["❌ Error\nLog and alert"]

    F --> H["Notification\nto Slack/email"]
    G --> I["Notification\nto PagerDuty"]

    style E fill:#e8f5e9,stroke:#2e7d32
    style F fill:#fff3e0,stroke:#e65100
    style G fill:#ffebee,stroke:#c62828

Summary #

• Drift is a normal condition that will always happen in an active team — what matters is detecting and responding to it quickly.
• terraform plan -refresh-only is the safest way to detect drift without risking infrastructure changes.
• Two options when drift is detected: apply to revert to the configuration, or update the configuration to reflect reality — choose based on context.
• ignore_changes for attributes truly managed outside Terraform — prevents false-positive drift detection.
• Scheduled automated drift detection (e.g. every working day at 8 AM) ensures the team learns about drift before it becomes a production problem.
• -detailed-exitcode enables CI/CD integration — exit code 2 means drift detected.
• Clear team policies about manual changes are the first line of defense — not a tool, but a habit.

← Previous: Idempotency   Next: What is a Resource? →