Locking #

Imagine two engineers running terraform apply simultaneously against the same state. Without locking, both processes could read the same state, make conflicting changes, and overwrite each other — producing state that’s inconsistent with the actual infrastructure condition. State locking prevents this by ensuring only one process can modify state at a time.

Why Locking Is Needed #

flowchart TD
    subgraph "Without Locking"
        A1["Engineer A\nterraform apply"] --> R1["Read state v1"]
        A2["Engineer B\nterraform apply"] --> R2["Read state v1"]
        R1 --> W1["Write state v2\n(create instance-X)"]
        R2 --> W2["Write state v3\n(create instance-Y)"]
        W1 -.->|"Overwritten"| LOST["Change A\nlost!"]
        W2 --> STATE3["State v3\nOnly instance-Y"]
    end

    subgraph "With Locking"
        B1["Engineer A\nterraform apply"] --> LOCK["LOCK acquired"]
        B2["Engineer B\nterraform apply"] --> WAIT["Wait for lock\nrelease..."]
        LOCK --> DONE1["Apply complete\nLock released"]
        DONE1 --> LOCK2["LOCK acquired\nby B"]
        LOCK2 --> DONE2["Apply complete\nLock released"]
    end

    style LOST fill:#ffebee,stroke:#c62828
    style STATE3 fill:#ffebee,stroke:#c62828
    style DONE2 fill:#e8f5e9,stroke:#2e7d32

WITHOUT LOCKING — WHAT CAN HAPPEN:

  T+0s  Engineer A: terraform apply (reads state, plans +3 resources)
  T+1s  Engineer B: terraform apply (reads the SAME state, plans +2 resources)
  T+10s Engineer A: apply done, writes state with 3 new resources
  T+12s Engineer B: apply done, writes state with 2 new resources
  T+12s → State now only has B's 2 resources, A's 3 resources MISSING from state
  T+12s → A's resources still exist in the cloud but Terraform doesn't know
  T+12s → Terraform state is inconsistent with reality

WITH LOCKING:

  T+0s  Engineer A: terraform apply → LOCK acquired ✅
  T+1s  Engineer B: terraform apply → Error: state is locked ❌
  T+10s Engineer A: done → LOCK released
  T+11s Engineer B: terraform apply → LOCK acquired ✅ (state is now up to date)

How Locking Works #

State locking works on the same principle across all backends — only the lock storage mechanism differs.

flowchart TD
    A["terraform apply\nor terraform plan"] --> B["Try to acquire\nthe state lock"]
    B --> C{"Lock\nacquired?"}
    C -->|"Yes"| D["Read state\nCalculate changes\nApply"]
    D --> E["Write new state\nRelease lock"]
    C -->|"No"| F["Error:\nError acquiring the state lock"]
    F --> G{"Lock\nstale?"}
    G -->|"Yes"| H["terraform force-unlock\n<LOCK_ID>"]
    G -->|"No"| I["Wait or\ncancel the operation"]

    style E fill:#e8f5e9,stroke:#2e7d32
    style F fill:#ffebee,stroke:#c62828

Locking with the S3 Backend (DynamoDB) #

# S3 backend with DynamoDB locking
terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/network/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "terraform-locks"  # DynamoDB table for locking
    encrypt        = true
  }
}

# The required DynamoDB table
resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Error: State Locked #

# The error that appears when the state is locked by another process
$ terraform apply

Error: Error acquiring the state lock

Error message: ConditionalCheckFailedException: The conditional request
failed
Lock Info:
  ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
  Path:      my-terraform-state/prod/network/terraform.tfstate
  Operation: OperationTypeApply
  Who:       engineer@hostname
  Version:   1.7.0
  Created:   2024-01-15 10:30:00.000000000 +0000

Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again.

Force Unlock #

terraform force-unlock is used to release a lock left behind by a failed or crashed process. Use it with care.

# First check whether another process is truly finished/crashed
# DON'T force-unlock if another process is still running!

# Force unlock with the ID from the error message
terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890

# Force unlock with auto-approve (skip confirmation)
terraform force-unlock -force a1b2c3d4-e5f6-7890-abcd-ef1234567890

Be careful with force-unlock. Only use it when you’re sure no other Terraform process is running. Releasing a lock while another process is active can cause state corruption — a far worse problem than a leftover lock. Always coordinate with your team before force-unlocking.

Best Practices for Locking #

CI/CD PIPELINES:
  ✓ Make sure every pipeline uses a backend with active locking
  ✓ Don't run multiple pipelines against the same state in parallel
  ✓ Use sequential pipeline stages for the same environment
  ✓ Set timeouts on pipelines — a lock shouldn't be held too long

COLLABORATIVE TEAMS:
  ✓ Use a remote backend with locking (S3+DynamoDB, TF Cloud, etc.)
  ✓ There's no excuse for a local backend in a team of more than 1
  ✓ Communicate who's applying in the team channel
  ✓ Avoid terraform force-unlock without coordination

STALE LOCKS:
  ✓ A lock is considered stale if the process that took it has crashed
  ✓ Check whether a Terraform process is still running before force-unlocking
  ✓ In CI/CD, cancel pipelines with SIGINT (not SIGKILL)
    so Terraform can release the lock gracefully

Locking Best Practices #

# Always use locking for state in team environments
# S3 backend: must configure dynamodb_table
# GCS backend: built-in locking
# Terraform Cloud: built-in locking

# Configure a lock timeout for CI/CD
terraform apply -lock-timeout=5m

# Monitor lock usage in DynamoDB
aws dynamodb scan --table-name terraform-state-lock

# DynamoDB table for locking
resource "aws_dynamodb_table" "terraform_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Troubleshooting Lock Issues #

Lock issues are one of the most common problems faced in teams.

# Problem: "Error: Error acquiring the state lock"
# Meaning: another process is currently using the state

# 1. Check who holds the lock
aws dynamodb get-item \
  --table-name terraform-state-lock \
  --key '{"LockID":{"S":"my-bucket/env:/terraform.tfstate"}}'

# The output contains info about who locked it:
# {
#   "LockID": {"S": "my-bucket/env:/terraform.tfstate"},
#   "Info": {"S": "{\"who\":\"hostname\",\"version\":\"1.6.3\",\"created\":\"2024-01-15T10:30:00Z\"}"}
# }

# 2. If the lock is stuck (process is dead), force unlock
terraform force-unlock LOCK_ID
# LOCK_ID can be seen in the error message or DynamoDB

# 3. CAREFUL: force-unlocking while another process is running
# = state corruption! Make sure nobody is applying

# 4. Prevention: use a lock timeout
terraform apply -lock-timeout=30s

flowchart TD
    A["terraform apply"] --> B{"Lock\navailable?"}
    B -->|"Yes"| C["Acquire lock\n& proceed"]
    B -->|"No"| D["Wait\n(lock timeout)"]
    D --> E{"Timeout\nexpired?"}
    E -->|"Not yet"| B
    E -->|"Yes"| F["Error: cannot\nacquire lock"]
    F --> G{"Stuck\nlock?"}
    G -->|"Yes"| H["terraform\nforce-unlock"]
    G -->|"No"| I["Wait for the other\nprocess to finish"]

    style A fill:#e3f2fd,stroke:#1565c0
    style C fill:#e8f5e9,stroke:#2e7d32
    style F fill:#ffebee,stroke:#c62828
    style H fill:#fff3e0,stroke:#e65100

Lock Monitoring in Production #

# Script for monitoring lock status
#!/bin/bash
TABLE="terraform-state-lock"

echo "=== Active Terraform Locks ==="
aws dynamodb scan --table-name $TABLE --output table

# Alerting: send a notification if a lock lasts too long
LOCK_ITEMS=$(aws dynamodb scan --table-name $TABLE --output json)
LOCK_COUNT=$(echo $LOCK_ITEMS | jq '.Items | length')

if [ "$LOCK_COUNT" -gt 0 ]; then
  echo "WARNING: $LOCK_COUNT active lock(s) found!"
  # Send an alert to Slack/PagerDuty
fi

State Locking Internals #

Terraform uses conditional writes to ensure the atomicity of lock operations.

LOCK MECHANISM DETAIL:

1. terraform apply starts
2. Terraform creates a lock entry in the backend:
   - LockID: path to the state file
   - Info: JSON {who, version, created, path}
   - TTL: no expiry (until unlock)

3. All state operations check the lock
4. If a lock already exists → error
5. When finished → delete the lock entry

ATOMICITY GUARANTEE:
- S3 + DynamoDB backend: DynamoDB conditional put
- GCS backend: GCS preconditions
- Consul backend: Consul KV transactions
- All ensure only 1 writer

# A robust backend configuration
terraform {
  backend "s3" {
    bucket         = "terraform-state-prod"
    key            = "infrastructure/terraform.tfstate"
    region         = "ap-southeast-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
    
    # Recommended: use a specific KMS key
    kms_key_id = "arn:aws:kms:ap-southeast-1:123456:key/abc-123"
  }
}

Distributed Locking Patterns #

DISTRIBUTED LOCK PATTERN:

sequenceDiagram
    participant A as "Worker A"
    participant DB as "DynamoDB"
    participant B as "Worker B"

    A->>DB: AcquireLock
    DB-->>A: LockGranted
    B->>DB: AcquireLock
    DB-->>B: LockDenied
    Note over A: DoWork
    A->>DB: ReleaseLock
    B->>DB: AcquireLock
    DB-->>B: LockGranted

# Monitoring lock contention
# CloudWatch alarm for DynamoDB throttling
resource "aws_cloudwatch_metric_alarm" "lock_contention" {
  alarm_name          = "terraform-lock-contention"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ThrottledRequests"
  namespace           = "AWS/DynamoDB"
  period              = 300
  statistic           = "Sum"
  threshold           = 5
  alarm_description   = "Terraform state lock contention detected"
  dimensions = {
    TableName = "terraform-state-lock"
  }
}

# Diagnostic commands
# Check who holds the lock
aws dynamodb get-item   --table-name terraform-state-lock   --key '{"LockID":{"S":"bucket/key/terraform.tfstate"}}'

# Force unlock (CAREFUL!)
terraform force-unlock <LOCK_ID>
# Only use if you're 100% sure no other process is running

Summary #

• State locking prevents concurrent writes — only one process can modify state at a time, preventing state corruption.
• Remote backends generally include locking — S3+DynamoDB, GCS, Azure Blob, and Terraform Cloud all have built-in locking mechanisms.
• The “state is locked” error is a safety feature — not a bug. It means another process is running.
• Use terraform force-unlock with care — only after confirming no other process is active.
• CI/CD must be sequential for the same state — don’t run multiple pipelines operating on the same state simultaneously.
• Locking + a remote backend is the minimum standard for collaborating teams — the local backend has none of this protection.

← Previous: Lifecycle   Next: Local →