Breaking things on purpose: chaos engineering

Every system has weaknesses nobody knows about. A timeout that's too long. A retry that turns a small problem into a big one. A failover that worked once in testing two years ago and never since. Usually, you discover them during a real outage, at the worst possible time.

Chaos engineering is about finding weaknesses on purpose, in a controlled way, before they cause problems.

The idea

Netflix made chaos engineering famous with Chaos Monkey, a tool that randomly terminated production servers. The idea sounded crazy, but the logic was simple: if servers will fail anyway, make sure your system handles it, by making failure a normal, everyday event.

Today the practice is more careful and scientific than random destruction:

1. Define steady state: what does "working normally" look like? Usually measured with metrics like error rate, latency or orders per minute.
2. Form a hypothesis: "If one database replica fails, error rates stay below 0.5% and the app recovers within 30 seconds."
3. Introduce a failure: kill the replica.
4. Observe: did the steady state hold?
5. Learn and fix: if not, you've found a weakness before it caused a real outage.

Chaos engineering isn't about breaking things. It's about proving that your system can take a hit, and finding out where it can't.

What kinds of failures?

Common experiments:

• Kill pods or nodes: does Kubernetes reschedule correctly? Do clients reconnect?
• Add network latency: what happens when a dependency becomes slow, not down? (Often worse than being down.)
• Drop network traffic between services, or to an external API.
• Fill disks or exhaust memory and CPU.
• Fail a whole availability zone: does traffic shift?
• Break DNS or certificates.
• Make a dependency return errors.

Your first game day

A game day is a planned session where a team runs chaos experiments together. It's the best way to start.

How I'd run one:

1. Pick a non-production environment that's close to production, like staging.
2. Choose one service and one or two experiments. For example: kill one pod of the checkout service, then add 500 ms latency to its database calls.
3. Write down the hypothesis for each experiment before starting.
4. Get everyone in a room or call: the service owners, someone watching dashboards, someone running the experiment.
5. Run the experiment, watch the metrics, take notes.
6. Have a stop button: know how to end the experiment immediately.
7. Debrief: what did we expect, what happened, what will we fix?

The first game day often shows something unexpected: a missing alert, a dashboard that's confusing, an outdated runbook, or a retry storm. That alone makes it worth doing.

Tools to try

• Chaos Mesh (CNCF): a Kubernetes-native tool with a nice UI. Experiments are Kubernetes resources: pod kill, network delay, stress, I/O faults, time skew and more.
• LitmusChaos (CNCF): Kubernetes chaos with a large hub of ready-made experiments and workflows.
• AWS Fault Injection Service and Azure Chaos Studio: managed services for injecting failures into cloud resources, including AZ-level disruptions.
• Gremlin: a commercial platform with safety controls and a good UI.
• Toxiproxy: a simple proxy for simulating network conditions in tests.

A Chaos Mesh experiment adding latency:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: checkout-db-latency
  namespace: shop
spec:
  action: delay
  mode: all
  selector:
    namespaces: ["shop"]
    labelSelectors:
      app: checkout
  delay:
    latency: "500ms"
    jitter: "100ms"
  direction: to
  target:
    mode: all
    selector:
      labelSelectors:
        app: orders-db
  duration: "5m"

Starting small, staying safe

• Start in staging, with small blast radius: one pod, one service.
• Make experiments reversible and time-limited.
• Monitor closely and abort if things go further than expected.
• Tell people before you run experiments, especially at first.
• Move to production carefully, only once you trust your monitoring and rollback, and start with the least risky experiments during working hours.

Did it help?

Measure the effect over time:

• Number of weaknesses found and fixed.
• Incidents caused by failure modes you'd already tested (should go down).
• Recovery time in real incidents.
• Team confidence during incidents, which is hard to measure but very real.

So, should you try it?

Yes, even on a small scale. One game day on staging, with one simple experiment, teaches a team more about their system than months of reading architecture diagrams.

My take: every system has a breaking point. Better to find it on a calm Tuesday afternoon than on a busy Saturday night.

— Desly