Releasing without fear: canary, blue-green and feature flags

There's a particular kind of silence in a team chat right after a big production deployment. Everyone's watching dashboards. Nobody wants to be the first to say "is it just me, or are errors going up?"

Progressive delivery replaces that silence with a plan: release to a few users first, track what happens, and only then decide whether to continue. If something's wrong, only a small group notices, and the system can undo the change automatically.

The strategies compared

Rolling update. Kubernetes' default: replace pods a few at a time. Simple, but once the rollout starts, new and old versions mix and you have little control over who sees what.

Blue-green. Run the new version (green) next to the old one (blue). Test green, then switch all traffic at once. Rolling back is just switching back. The cost: double resources during the switch, and a "big bang" moment for users.

Canary. Send a small percentage of traffic to the new version, for example 5%, then 25%, then 50%, then 100%, checking metrics at each step. Slower, but the safest way to catch problems that only appear under real traffic.

Feature flags. Deploy code with the new feature switched off, then turn it on for specific users, a percentage, or everyone, without deploying again. This separates deploying (getting code into production) from releasing (letting users see it).

	Rolling	Blue-green	Canary	Feature flags
Control over exposure	Low	All or nothing	Gradual by traffic	Very fine, by user
Rollback speed	Slow	Instant	Fast	Instant
Extra infrastructure	None	Double during switch	Some	Flag service
Best for	Low-risk changes	Big switches	Risky backend changes	Product features

Deploying should be boring. Releasing should be a decision.

Argo Rollouts in practice

On Kubernetes, Argo Rollouts (and Flagger, its counterpart in the Flux ecosystem) adds canary and blue-green strategies with automatic analysis.

A Rollout replaces a Deployment:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: shop-api
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: { duration: 5m }
      - analysis:
          templates:
          - templateName: error-rate
      - setWeight: 25
      - pause: { duration: 10m }
      - setWeight: 50
      - pause: { duration: 10m }
  selector:
    matchLabels: { app: shop-api }
  template:
    # same pod template as a Deployment

The analysis step runs a Prometheus query, for example the error rate of the canary, and compares it with a threshold:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
  - name: error-rate
    interval: 1m
    count: 5
    successCondition: result[0] < 0.01
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(http_requests_total{app="shop-api",version="canary",code=~"5.."}[2m]))
          /
          sum(rate(http_requests_total{app="shop-api",version="canary"}[2m]))

If the error rate goes above 1%, the rollout is aborted and traffic goes back to the stable version. Nobody had to be awake.

For precise traffic percentages, Rollouts integrates with Gateway API implementations, service meshes and ingress controllers. Without them, it approximates weights using replica counts.

Feature flags with OpenFeature

Feature flags used to mean either a homemade if statement reading an environment variable, or a commercial service with its own SDK everywhere. OpenFeature, a CNCF project, standardises the API: your code uses one SDK, and you can plug in different flag providers behind it (flagd, LaunchDarkly, Flagsmith, Unleash, GrowthBook and others).

enabled, _ := client.BooleanValue(ctx, "new-checkout", false, evalCtx)
if enabled {
    return newCheckout(w, r)
}
return oldCheckout(w, r)

Good flag habits:

• Give every flag an owner and an expiry date. Old flags become technical debt fast.
• Clean up once a feature is fully released.
• Keep flags out of critical paths where the flag service being down would break things, or use sensible defaults.

Automatic rollback on SLO breach

The best setups link releases to your service level objectives. If a new version uses up the error budget faster than normal, the rollout stops and reverses. That turns "watch the dashboards nervously" into an automatic safety net based on what users actually experience.

So, which should you use?

Start with feature flags for product changes, they're the easiest win. Add canary releases with automatic analysis for your most important services. Use blue-green when you need a clean switch, like a major version change. And keep rolling updates for everything low-risk.

My take: release in small steps, let metrics make the first decision, and make sure going back is always one click away.

— Desly