Servers you never patch: golden images with Packer

There's an old way to run servers that many of us grew up with. You set up a machine, install your software, and then keep it alive for years. Patch it, tweak it, fix it by hand when something breaks. Over time, every server becomes slightly different, and nobody knows exactly what's on them. People call those servers "pets".

You never change a running server. When something needs to change, you build a new image, start new servers from it, and throw the old ones away.

Pets vs cattle, revisited

The famous analogy:

• Pets have names, are unique, and get nursed back to health when sick.
• Cattle have numbers, are identical, and get replaced when sick.

Containers made the cattle model normal for applications. But plenty of workloads still run on virtual machines: Kubernetes nodes themselves, databases, legacy apps, CI runners, appliances. Golden images bring the same idea to VMs.

What a golden image is

A golden image is a pre-built machine image (an AMI on AWS, a managed image on Azure, a custom image on Google Cloud, a template on VMware or Proxmox) that already contains:

• An up-to-date, hardened operating system
• Your monitoring and logging agents
• Security tools and configuration
• Sometimes the application itself

A new server starts from it in seconds, already configured. No long setup scripts at boot, no drift between machines.

If a server needs fixing, don't fix it. Fix the image, and replace the server.

Building images with Packer

Packer, from HashiCorp, is the most common tool for building images. You describe the image in HCL, and Packer starts a temporary machine, runs your setup steps, and saves the result as an image.

packer {
  required_plugins {
    amazon = {
      source  = "github.com/hashicorp/amazon"
      version = "~> 1.3"
    }
  }
}

source "amazon-ebs" "base" {
  region        = "eu-central-1"
  instance_type = "t3.small"
  ssh_username  = "ubuntu"
  ami_name      = "acme-base-{{timestamp}}"

  source_ami_filter {
    filters = {
      name                = "ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"
      virtualization-type = "hvm"
    }
    owners      = ["099720109477"] # Canonical
    most_recent = true
  }
}

build {
  sources = ["source.amazon-ebs.base"]

  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get -y upgrade",
      "sudo apt-get -y install unattended-upgrades auditd"
    ]
  }

  provisioner "ansible" {
    playbook_file = "./playbooks/harden.yml"
  }
}

The same template can build images for several clouds at once. Provisioners can be shell scripts, Ansible playbooks, or other tools.

Image pipelines

Building images by hand defeats the purpose. Put it in a pipeline:

1. Build the image on a schedule (weekly is common) and whenever the template changes.
2. Test it: boot an instance, run checks (services running, ports closed, hardening applied). Tools like Goss or InSpec help.
3. Scan it for vulnerabilities.
4. Publish it, tagged with a version and date, and share it with the accounts that need it.
5. Roll it out by updating the image ID in your infrastructure code.
6. Clean up old images after a retention period.

Cloud providers also offer managed services for this, like EC2 Image Builder on AWS and Azure VM Image Builder.

How often I rebuild

Security patches are released constantly. With immutable infrastructure, patching means rebuilding the image and replacing servers.

A weekly rebuild keeps images close to current. For critical vulnerabilities, trigger an extra build right away. The beauty of this model: patching is the same routine as any other change, fully automated and tested, instead of a stressful manual patch night.

Pairing with autoscaling

Golden images work best with autoscaling groups (or VM scale sets, or managed instance groups):

1. Build a new image.
2. Update the launch template with the new image ID.
3. Trigger a rolling instance refresh: the group replaces old instances with new ones, a few at a time, waiting for health checks.
4. If something's wrong, roll back to the previous launch template version.

Kubernetes node pools work similarly: managed node groups can be updated to a new image with a rolling replacement. Tools like Karpenter go even further, regularly replacing nodes to keep them fresh.

What to watch out for

• Don't bake secrets into images. Fetch them at boot from a secret manager.
• Keep environment-specific config out of the image too. Use instance metadata, tags or a config service at startup.
• Stateful data must live on separate volumes or services, so replacing a server doesn't lose anything.
• Image sprawl: set a retention policy and clean up old images regularly.

So, is it worth it?

For any fleet of VMs that you run for more than a few weeks, yes. Fewer surprises, faster scaling, easier patching, and you always know exactly what's running.

My take: stop nursing servers. Build them right once, replace them often, and let the image carry the knowledge.

— Desly