Credential Rotation Strategy #

A credential that’s never rotated is a credential that will eventually be compromised without anyone noticing — or if someone does notice, it’s already too late. In the Terraform ecosystem, there are two types of credentials needing a rotation strategy: credentials used by Terraform itself (provider credentials), and credentials managed by Terraform for applications (database passwords, API keys). Both types have different rotation methods, and Terraform can either help or hinder this process depending on how you design it.

flowchart LR
    A["Provider\nCredentials\n(OIDC/IAM)"] --> B["Terraform"]
    C["App\nSecrets\n(RDS/API key)"] --> B
    B --> D["Rotate\nProvider"] & E["Rotate\nApp Secrets"]

    style A fill:#3b82f6,stroke:#1e40af,color:#fff
    style C fill:#f59e0b,stroke:#d97706,color:#fff
    style B fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style D fill:#10b981,stroke:#059669,color:#fff
    style E fill:#10b981,stroke:#059669,color:#fff

Rotating Provider Credentials (CI/CD) #

The credentials used by Terraform and CI/CD pipelines to access the cloud are the most critical to rotate regularly.

ROTATION STRATEGIES BY CREDENTIAL TYPE:

OIDC Tokens (GitHub Actions, GitLab CI):
  - No manual rotation needed
  - Tokens are generated fresh on every pipeline run
  - Expire automatically within 1 hour
  ✓ This method has no credentials needing rotation

AWS SSO / IAM Identity Center:
  - Session tokens expire automatically (8-12 hours default)
  - Admins can revoke sessions anytime
  - Effective rotation only requires changing the SSO assignment
  ✓ Far easier than rotating static credentials

Static Access Keys (IAM User):
  - Need regular rotation (at least every 90 days)
  - Manual rotation: create a new key → update the secret → delete the old key
  - Prone to human error and downtime during rotation
  ✗ Avoid using this in CI/CD

# If you still use static credentials, here's the safe rotation method:

# 1. Create a new access key (one IAM user can have 2 keys at once)
aws iam create-access-key --user-name terraform-ci-user
# Save the new key, don't delete the old key yet

# 2. Update the secret in GitHub Actions
gh secret set AWS_ACCESS_KEY_ID --body "NEWKEYID"
gh secret set AWS_SECRET_ACCESS_KEY --body "NEWSECRETKEY"

# 3. Verify the pipeline runs with the new credentials
# Run the pipeline and make sure it succeeds

# 4. Deactivate the old key first (don't delete yet — you can still roll back)
aws iam update-access-key --access-key-id OLDKEYID --status Inactive --user-name terraform-ci-user

# 5. Once confident the new key works, delete the old key
aws iam delete-access-key --access-key-id OLDKEYID --user-name terraform-ci-user

Zero-Downtime Database Password Rotation #

Database password rotation is one of the trickiest because running applications use the old password, and during rotation there’s a window where the new password hasn’t reached all applications yet.

THE DATABASE PASSWORD ROTATION PROBLEM:

  Condition before rotation:
  Old password: "OldP4ssword"
  Application A: uses "OldP4ssword" ← running normally
  Application B: uses "OldP4ssword" ← running normally

  During rotation:
  Step 1: terraform apply — changes the DB password to "NewP4ssword"
    → All connections with the old password FAIL
    → Applications A and B: connection error!

  Step 2: Update the Kubernetes secret / SSM Parameter
    → Requires a pod restart for applications to get the new password
    → There's downtime during the restart

  This is the dangerous "big bang rotation" problem.

SOLUTION: DUAL-PASSWORD ROTATION (Zero Downtime)

  AWS Secrets Manager supports this natively:

  Phase 1: Create the new secret
    A new password is created, but the old password still works on the DB
    Applications still use the old password

  Phase 2: Set pending
    The new password exists in Secrets Manager
    Applications aren't using it yet

  Phase 3: Test the new secret
    Verify the new password can connect to the DB

  Phase 4: Finish rotation
    The old password is deactivated on the DB
    Applications refreshing credentials will get the new password

  Key: Applications must support dynamic credential refresh
  (read from Secrets Manager on every request, not just at startup)

# Configuring automatic rotation in AWS Secrets Manager via Terraform

resource "aws_secretsmanager_secret" "db_password" {
  name = "prod/database/master-password"

  # Automatic rotation every 30 days
  rotation_rules {
    automatically_after_days = 30
  }
}

resource "aws_secretsmanager_secret_rotation" "db_rotation" {
  secret_id           = aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.db_rotation.arn

  rotation_rules {
    automatically_after_days = 30
  }
}

# For RDS, use the managed rotation provided by AWS
# (easier than a custom Lambda)
resource "aws_db_instance" "main" {
  identifier = "production-db"
  # ...

  manage_master_user_password   = true
  master_user_secret_kms_key_id = aws_kms_key.rds.arn
}

resource "aws_secretsmanager_secret_rotation" "rds_rotation" {
  secret_id           = aws_db_instance.main.master_user_secret[0].secret_arn
  rotate_immediately  = false

  rotation_rules {
    automatically_after_days = 30
    # Or use schedule_expression for rotation at a specific time:
    # schedule_expression = "cron(0 2 1 * ? *)"  # 2am, 1st of every month
  }
}

Rotating API Keys for Third-Party Services #

# Patterns for rotating API keys used by Terraform
# (for example: Cloudflare API tokens, Datadog API keys)

# ANTI-PATTERN: The API key directly in the provider
provider "cloudflare" {
  api_token = "secret-token-that-never-rotates"
}

# CORRECT: Read from Secrets Manager, the token can rotate without config changes
data "aws_secretsmanager_secret_version" "cloudflare_token" {
  secret_id = "prod/cloudflare/api-token"
}

provider "cloudflare" {
  api_token = jsondecode(data.aws_secretsmanager_secret_version.cloudflare_token.secret_string)["token"]
}

# Rotation method:
# 1. Create a new API token in Cloudflare
# 2. Update the secret in Secrets Manager
#    aws secretsmanager put-secret-value --secret-id prod/cloudflare/api-token --secret-string '{"token":"newtoken"}'
# 3. Run terraform plan/apply — it will use the new token
# 4. Delete the old token in Cloudflare after verification succeeds

Preventing Unnecessary Rotation via ignore_changes #

Terraform needs to know that some credentials will change outside Terraform (rotated by Secrets Manager or externally), and it shouldn’t try to “restore” them to the old value.

resource "aws_db_instance" "main" {
  identifier = "production-db"
  username   = "admin"
  password   = var.initial_db_password

  lifecycle {
    # Ignore password changes — the password is rotated outside Terraform
    # by the Secrets Manager rotation Lambda
    ignore_changes = [password]
  }
}

# With ignore_changes, Terraform won't:
# - Detect the password change as drift
# - Try to restore the password to its initial value at plan time
# The password is fully managed outside Terraform after initial setup

flowchart TD
    A["Old credentials"] --> B["Generate new"]
    B --> C["Update config"]
    C --> D["Test access"]
    D --> E["Revoke old"]
    E --> F["Done"]

    style A fill:#ef4444,stroke:#dc2626,color:#fff
    style B fill:#3b82f6,stroke:#1e40af,color:#fff
    style C fill:#f59e0b,stroke:#d97706,color:#fff
    style D fill:#8b5cf6,stroke:#6d28d9,color:#fff
    style E fill:#10b981,stroke:#059669,color:#fff
    style F fill:#10b981,stroke:#059669,color:#fff

Automated Credential Rotation #

Manual rotation is prone to human error. Automating credential rotation is highly recommended.

# AWS: Rotating access keys with Lambda
resource "aws_iam_user" "terraform_ci" {
  name = "terraform-ci-user"
}

# AWS Secrets Manager to store the credential
resource "aws_secretsmanager_secret" "ci_creds" {
  name        = "terraform-ci-credentials"
  description = "CI/CD credentials for Terraform"
}

# Rotation schedule
resource "aws_secretsmanager_secret_rotation" "ci_creds" {
  secret_id           = aws_secretsmanager_secret.ci_creds.id
  rotation_lambda_arn = aws_lambda_function.rotate_creds.arn

  rotation_rules {
    automatically_after_days = 90  # Rotate every 90 days
  }
}

flowchart TD
    A["Secret Manager\n(rotation schedule)"] --> B["Trigger\nLambda"]
    B --> C["Generate\nnew credentials"]
    C --> D["Update the\nIAM user"]
    D --> E["Update the\nsecret value"]
    E --> F["Verify the\nnew creds work"]
    F --> G["Deactivate\nold credentials"]

    style A fill:#e3f2fd,stroke:#1565c0
    style G fill:#ffebee,stroke:#c62828

Credential Rotation Monitoring #

Credential rotation must be monitored to make sure it succeeds.

# Monitor credential age
aws iam list-access-keys --user-name terraform-ci \
  --query 'AccessKeyMetadata[].[AccessKeyId,Status,CreateDate]' \
  --output table

# Alert if a credential is older than 90 days
KEY_AGE=$(aws iam list-access-keys --user-name terraform-ci \
  --query 'AccessKeyMetadata[0].CreateDate' --output text)

DAYS_OLD=$(( ($(date +%s) - $(date -d "$KEY_AGE" +%s)) / 86400 ))

if [ "$DAYS_OLD" -gt 90 ]; then
  echo "WARNING: Access key is $DAYS_OLD days old!"
  # Send an alert
fi

flowchart TD
    A["Scheduled\nCheck"] --> B{"Credential\nage > 90 days?"}
    B -->|"Yes"| C["Alert:\nRotate now!"]
    B -->|"No"| D["OK ✅"]
    C --> E["Auto-rotate\n(Lambda)"]
    E --> F["Verify\nnew creds"]
    F --> G["Deactivate\nold creds"]

    style C fill:#fff3e0,stroke:#e65100
    style D fill:#e8f5e9,stroke:#2e7d32
    style G fill:#ffebee,stroke:#c62828

IAM Credential Rotation Policy #

# IAM policy to enforce credential rotation
resource "aws_iam_policy" "credential_rotation" {
  name = "credential-rotation-policy"
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = [
          "secretsmanager:GetSecretValue",
          "secretsmanager:RotateSecret"
        ]
        Resource = aws_secretsmanager_secret.ci_creds.arn
      }
    ]
  })
}

# Lambda function for auto-rotation
resource "aws_lambda_function" "rotate_creds" {
  filename      = "rotate-creds.zip"
  function_name = "rotate-terraform-creds"
  role          = aws_iam_role.lambda.arn
  handler       = "index.handler"
  runtime       = "python3.11"
  timeout       = 300
}

Summary #

  • OIDC is the best approach for CI/CD — tokens are generated fresh on every run and expire automatically, nothing needs manual rotation.
  • Static credential rotation needs two steps: create a new key → update the pipeline → verify → deactivate the old key → delete. Don’t delete without verification.
  • “Big bang rotation” of database passwords causes downtime — use dual-password rotation via AWS Secrets Manager’s managed rotation, natively supported by RDS.
  • Automatic rotation in Secrets Manager is the best solution for database and service credentials — AWS manages the Lambda rotation function and timeline.
  • ignore_changes = [password] in the lifecycle block prevents Terraform from trying to “restore” a password already rotated by Secrets Manager.
  • Read credentials from Secrets Manager in the provider config instead of hardcoding — this allows rotating provider credentials without changing the Terraform configuration.

← Previous: Environment & Secret   Next: Secret Exposure Risk →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact