Manual Infrastructure #

Imagine joining a team that runs 40 servers in production. There’s no documentation about how those servers were created. The only thing that exists is a senior engineer who “knows” the configuration by heart. One day that person resigns, and the team realizes nobody knows exactly what settings are used on each server. Some servers run different software versions, network configuration is inconsistent, and no one can guarantee that staging truly replicates production. This is the reality of manually managed infrastructure — an approach that looks fast at first, but leaves behind expensive and dangerous technical debt.

This article dives deep into why manual infrastructure management is a problem — from snowflake servers and configuration drift to slow disaster recovery. You’ll also see how each of these problems is connected and amplifies the others, and why moving to Infrastructure as Code isn’t just a technical choice but an operational necessity.

What Manual Infrastructure Means #

Manual infrastructure is any approach where cloud resources are created, configured, and managed through human actions that aren’t automated or documented in code. This includes clicking around the AWS console, running ad-hoc CLI commands, or even writing scripts that only one person knows how to run.

flowchart LR
    subgraph Manual["Ways of Managing Infrastructure Manually"]
        A["🖱️ Clicking in\nthe Cloud Console"] --> B["💻 Running\nCLI Commands"]
        B --> C["📝 Writing\nAd-hoc Scripts"]
        C --> D["🧠 Only 1 Person\nKnows It"]
    end

    D --> E["⚠️ Problem:\nUndocumented"]
    D --> F["⚠️ Problem:\nNot Reusable"]
    D --> G["⚠️ Problem:\nUntested"]

    style E fill:#ffebee,stroke:#c62828
    style F fill:#ffebee,stroke:#c62828
    style G fill:#ffebee,stroke:#c62828

Every manual method shares the same fundamental problem: the process isn’t defined in a form that machines can verify. When you create a VPC through the console, nobody knows whether the subnet CIDR you picked is correct except you. When you run aws ec2 run-instances in a terminal, nobody checks whether the security group you used meets the team’s security standards.

Real Example: Creating an EC2 Instance Manually #

Here’s an example of how someone creates an EC2 instance through the AWS CLI manually. Notice how this process requires many steps that must be remembered and executed in the right order.

# Step 1: Create a VPC
VPC_ID=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 \
  --query 'Vpc.VpcId' --output text)

# Step 2: Create a Subnet
SUBNET_ID=$(aws ec2 create-subnet --vpc-id $VPC_ID \
  --cidr-block 10.0.1.0/24 \
  --query 'Subnet.SubnetId' --output text)

# Step 3: Create an Internet Gateway
IGW_ID=$(aws ec2 create-internet-gateway \
  --query 'InternetGateway.InternetGatewayId' --output text)
aws ec2 attach-internet-gateway --internet-gateway-id $IGW_ID \
  --vpc-id $VPC_ID

# Step 4: Create a Route Table
RT_ID=$(aws ec2 create-route-table --vpc-id $VPC_ID \
  --query 'RouteTable.RouteTableId' --output text)
aws ec2 create-route --route-table-id $RT_ID \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $IGW_ID
aws ec2 associate-route-table --route-table-id $RT_ID \
  --subnet-id $SUBNET_ID

# Step 5: Create a Security Group
SG_ID=$(aws ec2 create-security-group \
  --group-name "web-sg" --description "Web SG" --vpc-id $VPC_ID \
  --query 'GroupId' --output text)
aws ec2 authorize-security-group-ingress --group-id $SG_ID \
  --protocol tcp --port 443 --cidr 0.0.0.0/0

# Step 6: Launch the Instance
INSTANCE_ID=$(aws ec2 run-instances \
  --image-id ami-0abcdef1234567890 \
  --instance-type t3.micro \
  --subnet-id $SUBNET_ID \
  --security-group-ids $SG_ID \
  --query 'Instances[0].InstanceId' --output text)

# ...and many more steps after this

Imagine having to memorize and run all of these steps every time you need a new instance. Now imagine five people on the team each doing the same thing — slightly differently every time. That’s where the problems begin.

Anti-Pattern: Equating “It Works” with “It’s Correct” #

ANTI-PATTERN:
  ✗ "The server works fine, why automate it?"
  ✗ "When I created it, I just clicked through the console, done in 30 minutes"
  ✗ "I already have a bash script, just run it"

WHY IT'S DANGEROUS:
  ✓ "Works" ≠ "documented" ≠ "repeatable" ≠ "safe"
  ✓ A 30-minute process can become 3 days if the person who built it resigns
  ✓ A bash script without idempotency can create duplicate resources

The statement “it works” is the most dangerous trap in infrastructure operations. A server running today might not be recreatable tomorrow because nobody knows its full configuration. This isn’t a technical problem — it’s an operational one that can affect the business.

Snowflake Servers #

A snowflake server is a unique server — no other server has exactly the same configuration. It’s called a “snowflake” because like snow, each server looks similar at a glance but no two are identical in detail. The problem is that this uniqueness isn’t intentional — it’s the accumulation of manual changes made over months or years.

flowchart TD
    A["Server Created\n(all identical)"] --> B["Admin A\ninstalls package X"]
    A --> C["Admin B\nchanges config Y"]
    A --> D["Admin C\napplies security patch Z"]

    B --> E["Server 1\n(package X + default config + no patch)"]
    C --> F["Server 2\n(no package X + config Y + no patch)"]
    D --> G["Server 3\n(no package X + default config + patch Z)"]

    E --> H["❓ Which one is correct?\nNobody knows"]
    F --> H
    G --> H

    style H fill:#ffebee,stroke:#c62828

In the scenario above, each admin makes a change that’s “correct” according to their own understanding. But without a single source of truth, nobody can confirm which configuration should be used across all servers.

Why Snowflake Servers Are Dangerous #

Snowflake servers create several interconnected problems:

Debugging becomes extremely difficult. When an error happens in production, the first troubleshooting step is finding the difference between the environment that works and the one that doesn’t. If every server is unique, you have no baseline to compare against. A bug that appears on server 1 might not appear on server 2 because of a configuration difference you’re not aware of.

Scaling becomes unpredictable. Adding a new server means trying to reproduce the configuration of an existing server — but that configuration is undocumented. The new server might run a different library version, a different timezone setting, or different JVM parameters, and these small differences can cause significant behavioral differences.

Compliance becomes impossible. Security standards require that all servers meet the same security baseline. With snowflake servers, there’s no way to verify that all servers are patched, follow the same hardening standards, or use approved software versions.

CHECKLIST: Does Your Team Have Snowflake Servers?
  □ Is there a server that "shouldn't be touched because it's been running for ages"?
  □ Are there OS/package version differences between servers that should be identical?
  □ Are you unsure you could rebuild a production server from scratch?
  □ Is there configuration known by only 1-2 people?
  □ Does every incident have a different root cause?

  If you answered "yes" to 2 or more, your team most likely
  has snowflake servers.

Configuration Drift #

Configuration drift happens when the actual state of your infrastructure deviates from the expected state over time. Drift occurs gradually — one small change here, one hotfix there — until one day you realize production looks very different from what you thought it was.

flowchart TD
    A["Initial Infrastructure\n(as designed)"] --> B["Time Passes..."]

    B --> C["Change 1:\nAdmin adds a security group rule\nto troubleshoot"]
    B --> D["Change 2:\nDevOps team resizes an instance\nwithout updating documentation"]
    B --> E["Change 3:\nHotfix: open port 8080\nin production temporarily"]
    B --> F["Change 4:\nChange subnet CIDR\nto accommodate new resources"]

    C --> G["Actual State:\nVery different from the original design"]
    D --> G
    E --> G
    F --> G

    G --> H["Nobody knows which changes were intentional\nand which were accidents"]

    style H fill:#ffebee,stroke:#c62828

The Impact of Configuration Drift #

Configuration drift isn’t a theoretical problem. Here are real scenarios that happen often in production:

Scenario 1: An open security group. A developer opens port 22 from 0.0.0.0/0 for troubleshooting on a Friday night. He plans to close it Monday morning but forgets. Three months later, a security audit finds the production SSH port exposed to the public — and nobody knows when or why it happened.

Scenario 2: A different instance type. The team runs load testing and changes the instance type from t3.medium to t3.2xlarge. After testing finishes, they forget to revert it. The next AWS bill balloons 8x, and it takes 2 days to find the cause because no change was recorded in Git.

Scenario 3: A manual fix that wasn’t replicated. An admin fixes a DNS problem on one production server by editing /etc/resolv.conf directly. When a new server is added via an auto scaling group, it doesn’t have the same fix and experiences intermittent DNS failures that are very hard to debug.

DRIFT THAT OFTEN GOES UNNOTICED:

Network Infrastructure:
  ✗ Security group rules added/removed without documentation
  ✗ Route table modified for troubleshooting
  ✗ NACL rules changed temporarily and never restored

Compute:
  ✗ Instance type changed for testing, never reverted
  ✗ AMI updated manually on one instance but not others
  ✗ User data script modified directly on the instance

Storage:
  ✗ EBS volume resized without updating IaC
  ✗ S3 bucket policy changed directly in the console
  ✗ Lifecycle policy removed "temporarily"

Can’t Be Reproduced #

One of the most dangerous traits of manual infrastructure is the inability to reproduce an environment identically. When you can’t recreate the exact same infrastructure, you lose the ability to do disaster recovery, accurate testing, and consistent scaling.

// ANTI-PATTERN: relying on manual documentation for reproduction

// "Documentation" commonly found in teams without IaC:
// - A .txt file on Google Drive last updated 6 months ago
// - An internal wiki with incomplete content
// - A senior engineer's memory of the configuration
// - A Slack chat history you need to scroll through hundreds of messages

// CORRECT: configuration in code that's always accurate
resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.medium"
  subnet_id     = aws_subnet.private.id

  vpc_security_group_ids = [aws_sg.web.id]

  tags = {
    Name        = "web-server"
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

Disaster Scenario: Region Outage #

Imagine the following scenario: the ap-southeast-1 region suffers a massive outage. Your team has 30 servers there. With manual infrastructure, the recovery process is:

1. Try to remember or dig up the configuration of every server (hours)
2. Create the VPC, subnets, and security groups manually in the new region (hours)
3. Launch instances one by one with a best-guess configuration (hours)
4. Configure the load balancer, DNS, and monitoring (hours)
5. Test and debug configuration differences (days)

With Terraform, the same process:

# Change the region in one line
provider "aws" {
  region = "ap-northeast-1"  # Changed from ap-southeast-1
}

# Run a single command
# $ terraform apply
# The entire infrastructure is recreated in the new region
# in minutes, with identical configuration

The difference isn’t just about time — it’s about certainty. With manual work, you can’t be sure the configuration in the new region is identical to the old one. With Terraform, you have a guarantee that the same code produces the same infrastructure.

No Audit Trail #

An audit trail — a record of who changed what, when, and why — is a fundamental requirement for safe, compliant operations. Manual infrastructure almost always fails to provide this properly.

flowchart LR
    subgraph Manual["Manual Audit Trail"]
        M1["CloudTrail?"] --> M2["Yes, but scattered across\nthousands of events"]
        M2 --> M3["Who opened\nport 22? Who\nresized an instance?"]
        M3 --> M4["Impossible to trace\nwithout context"]
    end

    subgraph IaC["Audit Trail with IaC"]
        I1["Git History"] --> I2["Every change\nrecorded with a\ncommit message"]
        I2 --> I3["Pull Request =\napproval process"]
        I3 --> I4["Who, when, why\nand what changed —\nall clear"]
    end

    Manual --> X["❌ Not Compliant"]
    IaC --> Y["✅ Compliant"]

    style X fill:#ffebee,stroke:#c62828
    style Y fill:#e8f5e9,stroke:#2e7d32

Audit Problems in a Manual Environment #

When an auditor asks “who opened port 22 to the public in production?”, the team managing manual infrastructure has to:

1. Enable CloudTrail (if it isn’t already)
2. Filter through thousands of CloudTrail events
3. Match events to IAM users/roles
4. Try to connect events with business context

This process can take days and often doesn’t produce a satisfactory answer, because CloudTrail only records what happened at the API level, not why.

With Git as the audit trail, every change is recorded in a traceable commit:

# Audit trail with Git — clear and structured
$ git log --oneline --all security-groups.tf

a1b2c3d Open port 443 for the new load balancer (#234)
e4f5g6h Close port 22 from public — audit remediation (#198)
i7j8k9l Add SSH bastion access (#156)

# Every commit has:
# - Who (git author)
# - When (timestamp)
# - What changed (diff)
# - Why (commit message / PR description)

Undetected Human Error #

Humans make mistakes. That’s an unavoidable fact. The problem isn’t the mistake itself, but the absence of mechanisms to detect and prevent errors before they impact production.

# Examples of very common human errors:

# Typo in the CIDR block — meant /24, typed /16
aws ec2 create-subnet --vpc-id vpc-123 --cidr-block 10.0.0.0/16
# This error can create a CIDR overlap that's only
# discovered weeks later

# Forgot tags — untagged instances aren't monitored
aws ec2 run-instances --image-id ami-xxx --instance-type t3.micro
# Without an Environment tag, this instance isn't included in
# monitoring and cost allocation

# Wrong environment — deploying to production instead of staging
aws rds create-db-instance --db-instance-identifier app-prod ...
# Should have been app-staging. Without guardrails, this can
# overwrite the production database

Why Human Errors Often Go Undetected #

In a manual environment, there’s no automated mechanism checking whether a change was correct. Everything depends on manual review — which is also done by humans who might be tired, rushed, or lacking full context.

// ANTI-PATTERN: relying on manual review to prevent errors

// In a manual environment, "review" usually looks like this:
// 1. Someone changes the configuration in the console
// 2. They tell the team on Slack: "I opened port 8080 in prod"
// 3. The team replies: "ok" (without checking whether it's safe)
// 4. Nobody verifies whether the change meets the standards

// CORRECT: With Terraform, review happens on code in a Pull Request
// 1. The change is committed to a feature branch
// 2. The Pull Request automatically shows the diff
// 3. CI/CD runs terraform plan
// 4. The team reviews the plan + diff before approving
// 5. Changes can only be applied after approval

Slow Disaster Recovery #

Disaster recovery is the moment of truth for every infrastructure team. When a major failure happens — a region outage, a ransomware attack, a mass human error — how fast can you restore the infrastructure? The answer depends heavily on whether your infrastructure is managed manually or automated.

flowchart TD
    A["🚨 Disaster Happens\n(Region Outage)"] --> B{"How Is\nInfrastructure\nManaged?"}

    B -->|"Manual"| C["Manual Recovery Process"]
    B -->|"IaC (Terraform)"| D["Automated Recovery Process"]

    C --> C1["Look for documentation\n(might not exist)"]
    C1 --> C2["Reconstruct configuration\nfrom memory/documents"]
    C2 --> C3["Create infrastructure\none by one"]
    C3 --> C4["Test and debug\nconfiguration differences"]
    C4 --> C5["⏱️ Time: Days\nto weeks"]

    D --> D1["Change the region in\nthe configuration"]
    D1 --> D2["Run\fterraform apply"]
    D2 --> D3["New infrastructure\nidentical to the old one"]
    D3 --> D4["⏱️ Time: Minutes\nto hours"]

    style C5 fill:#ffebee,stroke:#c62828
    style D4 fill:#e8f5e9,stroke:#2e7d32

RTO and RPO #

Two key disaster recovery metrics:

• RTO (Recovery Time Objective): How fast the infrastructure must recover. With manual work, RTO can be days. With Terraform, RTO can be measured in minutes.
• RPO (Recovery Point Objective): How much data loss is acceptable. Manual infrastructure often lacks a consistent backup strategy, so RPO can be very large.

// Example: a backup strategy defined in Terraform
resource "aws_db_instance" "main" {
  identifier     = "app-database"
  engine         = "postgres"
  engine_version = "15.4"
  instance_class = "db.t3.medium"

  backup_retention_period = 7        # Keep backups for 7 days
  backup_window          = "03:00-04:00"
  multi_az               = true      # High availability

  # All backup parameters are defined in code
  # Nobody "forgets" to enable backups
}

With the configuration above, every time a database is created — whether in a new environment, a new region, or after a disaster — the backup parameters are always the same. There’s no risk of “forgetting to enable backups” or “forgetting to set multi-AZ”.

The Hidden Costs of Manual Infrastructure #

Many teams consider manual infrastructure “cheaper” because it doesn’t require investing time in writing IaC code. This calculation is wrong because it ignores the much larger hidden costs.

HIDDEN COSTS OF MANUAL INFRASTRUCTURE:

1. Engineer Time Cost
   ✗ Creating new infrastructure: hours (vs minutes with IaC)
   ✗ Troubleshooting environment differences: days
   ✗ Onboarding new engineers: weeks
   ✗ Documentation that's always outdated: ongoing cost

2. Downtime Cost
   ✗ Slow recovery = longer downtime
   ✗ Undetected human error = unexpected incidents
   ✗ Snowflake servers = longer troubleshooting

3. Compliance Cost
   ✗ Manual audits = days of engineer time
   ✗ Remediating audit findings = manual changes that can go wrong
   ✗ Inability to prove compliance = risk of fines

4. Opportunity Cost
   ✗ Engineers busy managing infrastructure = can't
     work on new features
   ✗ Slow deploys = slow feedback loop = slow iteration
   ✗ Fear of change = resistance to improvement

A Simple Calculation #

Imagine a DevOps engineer earning Rp 25 million/month spending 40% of their time on manual tasks that could be automated:

48 hours/month × Rp 156,000/hour = Rp 7.5 million/month wasted on work that could be automated. And that’s just one engineer — multiply that by team size.

From Manual to Infrastructure as Code #

The transition from manual infrastructure to IaC doesn’t have to happen all at once. A pragmatic, gradual approach is more realistic and lower risk.

flowchart LR
    A["Phase 1\nImport\nExisting\nResources"] --> B["Phase 2\nWrite IaC\nfor New\nResources"]
    B --> C["Phase 3\nRefactor\nOld Resources\ninto IaC"]
    C --> D["Phase 4\nFull IaC +\nCI/CD Pipeline"]

    A -.->|"terraform import"| B
    B -.->|"team policy"| C
    C -.->|"terraform plan\nreview"| D

    style A fill:#fff3e0,stroke:#e65100
    style B fill:#e3f2fd,stroke:#1565c0
    style C fill:#e8f5e9,stroke:#2e7d32
    style D fill:#f3e5f5,stroke:#6a1b9a

Step 1: Start with New Resources #

You don’t need to convert the entire infrastructure at once. Start with a policy: “all new resources must be created through Terraform.” Existing resources are left as they are — they’ll be migrated gradually later.

Step 2: Import Old Resources #

Terraform provides the terraform import command to bring existing resources into state management. This lets you manage old resources through code without deleting and recreating them.

# Import an existing EC2 instance into Terraform state
terraform import aws_instance.web i-0abc123def456789

# Import an existing S3 bucket
terraform import aws_s3_bucket.data my-existing-bucket

# After importing, write the configuration matching
# the resource's actual state

Step 3: Set Up Guardrails #

Once most resources are managed by Terraform, add guardrails to prevent manual changes:

GUARDRAILS YOU CAN APPLY:

Organizational:
  ✓ Policy: all infrastructure changes must go through a PR
  ✓ Mandatory review by at least 1 person before merging
  ✓ Process documentation in the README

Technical:
  ✓ IAM policies restricting who can change resources
  ✓ AWS Config Rules to detect manual changes
  ✓ CI/CD pipeline that automatically runs terraform plan
  ✓ Drift detection that alerts on changes outside Terraform

Summary #

• Manual infrastructure — whether clicking in the console, running CLI commands, or writing ad-hoc scripts — produces snowflake servers that are unique and can’t be reproduced.
• Configuration drift happens when the actual infrastructure state deviates from what’s expected, and it’s very hard to detect without automated tools.
• There’s no adequate audit trail — CloudTrail records API events but lacks business context, making compliance expensive and time-consuming.
• Human error goes undetected because there’s no automated validation mechanism before changes reach production.
• Disaster recovery is slow — without code defining the infrastructure, rebuilding requires manual reconstruction that takes days.
• The hidden costs of manual infrastructure far exceed the time investment of writing IaC — including engineer time, downtime, compliance, and opportunity cost.
• A gradual transition from manual to IaC is the realistic approach — start with new resources, import old ones, then add guardrails.

← Previous: What is Terraform?   Next: Imperative Tools →