Text size
On My Radar5 min read

AI writes the code. Who checks it?

AI-generated code brings new risks, including packages that don't exist. How to catch them in CI.

AI writes the code. Who checks it?
AI-generated editorial illustration

A growing share of the code in our repositories wasn't typed by a human. AI assistants write functions, tests, Dockerfiles and Terraform. Agents open whole pull requests. That's a huge productivity boost.

The increase in code volume means there's more to review, but the time available hasn't changed. Plus, AI-generated code has its own kind of mistakes. So the pipeline needs to do more checking.

The new risks in plain words

Insecure patterns that look fine. AI learned from public code, and a lot of public code is insecure. SQL built from strings, disabled TLS verification, overly broad permissions, secrets in config files.

Outdated advice. Deprecated APIs, old library versions with known vulnerabilities, Kubernetes manifests using removed API versions.

Hallucinated dependencies. The AI suggests importing a package that sounds right but doesn't exist.

License problems. Generated code that closely resembles code under a restrictive license.

Bigger changes, less review. When generating code is easy, pull requests get larger, and large pull requests get worse reviews.

"Slopsquatting": installing a package an AI invented

This one deserves its own section because it's new and sneaky.

Language models sometimes invent package names. Plausible ones, like requests-oauth-helper or express-secure-session. Researchers noticed that the same invented names come up repeatedly. So attackers can register those names on npm or PyPI and fill them with malware. When a developer, or an agent, trusts the suggestion and installs the package, the attacker's code runs.

It's a twist on typosquatting, and the community nicknamed it slopsquatting.

An AI suggesting a package isn't a recommendation. It's a guess, and attackers are betting on those guesses.

The checks I'd add to the pipeline

None of this needs exotic tools. It needs the normal DevSecOps toolbox, actually switched on and enforced:

  • Dependency checks. Software composition analysis (Dependabot alerts, OSV-Scanner, Snyk, Trivy) for known vulnerabilities. Plus tools like Socket that flag suspicious packages: brand new, few downloads, install scripts.
  • A policy for new dependencies. Adding a new package should be visible in the pull request and get a human look. Some teams only allow packages from an internal proxy or registry.
  • Static analysis (SAST). Semgrep, CodeQL or similar to catch injection, insecure crypto and dangerous functions.
  • Secret scanning on every push, with push protection where available.
  • IaC scanning. Checkov, Trivy or tflint for Terraform and Kubernetes manifests: open security groups, privileged containers, missing encryption.
  • License scanning if license compliance matters to your company.
  • Tests, actually required. The best defence against confident wrong code is a test that fails.

Reviewing when there's too much to review

Tools catch patterns. Humans still need to catch logic. A few habits help:

  • Keep pull requests small, even when AI makes big ones easy. Ask the agent to split work.
  • Mark AI-generated changes so reviewers know to look more carefully. Some teams use a label or commit trailer.
  • Review the tests first. If the tests are weak, the code passing them means little.
  • Ask "why" for every new dependency.
  • Don't let AI review its own code alone. AI review tools are a useful extra pair of eyes, not a replacement for a human approval.

So, who checks it?

The pipeline checks what it can automatically, and a human checks what really matters. That's how it's always worked. AI just makes it harder because there's more code, written faster, by something that never hesitates.

My take: AI can write the code. Your pipeline and your reviewers still have to earn the right to ship it.

— Desly