Rules as code: OPA, Kyverno and built-in policies

Every organisation has rules. Containers must not run as root. Every deployment needs an owner label. Images must come from our registry. Production needs at least two replicas. Usually these rules live in a wiki page, and they're followed exactly as well as wiki pages are read.

Policy as code turns those rules into something a computer enforces, on every change, without anyone having to remember them.

Why rules belong in code

• Consistent. The rule applies to everyone, every time, including at 3 a.m.
• Fast feedback. Developers learn about a problem the moment they try to deploy, with a clear message, not weeks later in an audit.
• Versioned and reviewed. Policies live in Git, go through pull requests, and have a history.
• Testable. You can write tests for your policies like any other code.
• Auditable. You can show exactly which rules apply and prove they're enforced.

Three ways to write them in Kubernetes

Kubernetes has admission control: every create or update request can be checked (validated) or modified (mutated) before it's stored. Policy engines plug in there.

Kyverno: policies as Kubernetes YAML

Kyverno, a CNCF project, lets you write policies as Kubernetes resources. If you can read a Deployment, you can read a Kyverno policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  rules:
  - name: require-image-tag
    match:
      any:
      - resources:
          kinds: ["Pod"]
    validate:
      message: "Images must use a specific version tag, not 'latest'."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

Kyverno can also mutate (add default labels or security settings), generate resources (create a default NetworkPolicy in every new namespace), and verify image signatures.

OPA Gatekeeper: policies in Rego

Open Policy Agent (OPA), also a CNCF project, is a general-purpose policy engine using a language called Rego. Gatekeeper integrates it with Kubernetes. Rego is very powerful and works far beyond Kubernetes (APIs, Terraform plans, CI pipelines), but it has a learning curve.

package k8srequiredlabels

violation[{"msg": msg}] {
  required := {"owner", "app"}
  provided := {label | input.review.object.metadata.labels[label]}
  missing := required - provided
  count(missing) > 0
  msg := sprintf("Missing required labels: %v", [missing])
}

ValidatingAdmissionPolicy: built into Kubernetes

Kubernetes now has its own policy mechanism, ValidatingAdmissionPolicy, stable since version 1.30. Rules are written in CEL (Common Expression Language) and run inside the API server, with no extra component to install:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-replicas-in-prod
spec:
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  validations:
  - expression: "object.spec.replicas >= 2"
    message: "Production deployments need at least 2 replicas."

(A ValidatingAdmissionPolicyBinding then applies it to specific namespaces.)

It's perfect for simple validation rules. For mutation, generation, image verification and reporting, Kyverno or Gatekeeper still offer more. Both engines are also adopting CEL themselves.

A rule in a wiki is a suggestion. A rule in an admission policy is a guarantee.

Policies I always add

A good starter set for most clusters:

• No latest image tags.
• Images only from approved registries.
• Required labels: owner, app, maybe cost-center.
• Resource requests required on every container.
• No privileged containers, no host networking, no host path mounts (outside system namespaces).
• Run as non-root.
• Liveness and readiness probes on deployments.
• Signed images in production.

Many of these are available ready-made in the Kyverno policy library and the Gatekeeper library.

Rolling out without breaking things

The fastest way to make enemies is to switch on twenty policies in enforce mode on a Friday. Instead:

1. Start in audit mode. Kyverno's Audit, Gatekeeper's dryrun or warn, or Warn/Audit actions for ValidatingAdmissionPolicy. Violations are reported, nothing is blocked.
2. Look at the reports. Kyverno creates PolicyReports, Gatekeeper records violations. Fix existing workloads.
3. Communicate. Tell teams which rules are coming, why, and how to comply.
4. Enforce gradually, rule by rule, namespace by namespace.
5. Allow documented exceptions, with an owner and an expiry date.

Testing policies

• Kyverno has a CLI (kyverno test) to run policies against sample resources.
• OPA has opa test for Rego unit tests.
• Run the same policies in CI against your manifests, so developers find out before they even reach the cluster.

So, which engine?

For most teams starting out, Kyverno is the easiest: YAML policies, a big library, mutation and image verification included. Use ValidatingAdmissionPolicy for simple rules where you don't want another component. Choose OPA when you want one policy language across Kubernetes, Terraform, APIs and more.

My take: write the rules down once, in code, and let the cluster remember them for you.

— Desly