AI agents in my pipeline: what actually works

In Will AI take my DevOps job? I wrote that AI is like a very fast junior engineer who never gets tired and never quite knows when it's wrong. I also said a new part of my job would be building guardrails for AI agents.

This post is the practical follow-up. Not "will AI change DevOps", but: which tasks can you hand to an agent today, and how do you set it up so a mistake stays small?

What changed

A year or two ago, AI in DevOps mostly meant autocomplete in the editor, or pasting an error into a chat window. Useful, but you were always in the driver's seat.

Agents are different. Tools like Claude Code, GitHub Copilot's coding agent, Cursor's background agents and others can take a task, read the repository, run commands, look at the results, try again, and open a pull request. They work in a loop, not a single answer.

That makes them much more useful, and much more important to control.

Tasks I'm comfortable handing to agents

The pattern is: well-defined, easy to verify, and reviewed before it reaches production.

Fixing a failing build. "The pipeline fails on this test, find out why and propose a fix." The agent reads the logs, reproduces the failure, and suggests a change. A human reviews the pull request.

Dependency upgrades that need code changes. Renovate opens the upgrade, the tests fail because an API changed, the agent updates the call sites.

Boring infrastructure changes. Adding a new environment variable across services, updating resource requests in Helm values, renaming a label everywhere.

Writing first drafts. A new pipeline, a Dockerfile, a Terraform module skeleton, a runbook from an incident timeline.

Explaining and documenting. "What does this old script do?" turned into a README.

The permissions I give them (and don't)

This is the part that matters most. My rules:

• Agents work on branches, never on main. Every change goes through a pull request and normal review.
• No production credentials. Ever. Agents run in CI or a sandbox with access to the repository and test environments only.
• Short-lived, narrow tokens. A GitHub token that can push to branches and open pull requests, not one that can change repository settings or approve its own PRs.
• Branch protection does the enforcement. Required reviews and required checks apply to agents exactly like they apply to people.
• Network limits where possible. An agent fixing a unit test doesn't need to reach the whole internet.
• Everything is logged. What the agent ran, what it changed, who started it.

Treat an AI agent like a new contractor on their first day: helpful, eager, and with exactly the access the task needs, nothing more.

Where they still fail

Agents are impressive, but I see the same failure patterns again and again:

• Making the test pass instead of fixing the bug. Sometimes the "fix" is changing the test's expectation. Read the diff carefully.
• Missing context. The agent doesn't know that setting exists because of an outage three years ago. It will happily "clean it up".
• Overly broad permissions in generated IAM policies or Kubernetes RBAC. * is always tempting.
• Prompt injection. An agent reading issues, logs or web pages can be tricked by text written to manipulate it. Never give an agent that reads untrusted input the power to do something irreversible.
• Confidence. It always sounds sure. That's not the same as being right.

My setup, step by step

If you want to try this on your team, here's a safe starting point:

1. Pick one repository with good test coverage.
2. Create a dedicated bot identity with branch-only push rights.
3. Run the agent in CI (or a container) triggered by a label or comment, for example adding agent:fix to a failing PR.
4. Give it a clear instructions file in the repo: how to build, how to test, what not to touch.
5. Require human review for every pull request it opens.
6. Track results for a month: how many PRs merged, how many needed rework, how many were wrong.

After a month you'll know where it actually saves time on your team, which beats any vendor benchmark.

So, is it worth it?

Yes, for the right tasks. Agents remove a lot of small, annoying work, and they're getting better quickly. But the value depends entirely on the guardrails around them. Good tests, good reviews and narrow permissions turn an agent from a risk into a very productive teammate.

My take: let agents propose, let pipelines check, and let humans decide. Same rule as before, now with real tools behind it.

— Desly