Profiling in production: where the CPU actually goes

Here's a familiar situation. A dashboard shows a service using way more CPU than it should. Latency is creeping up. You look at the metrics: yes, CPU is high. You look at the traces: requests are slow, but inside the service, not in its dependencies. You look at the logs: nothing unusual.

What you really want to know is: which code is using all that CPU? That's what profiling answers. And continuous profiling answers it all the time, in production.

What profiling shows

A profile records where a program spends its resources, down to individual functions and lines:

• CPU profiles: which functions are running on the CPU, and how much.
• Memory (heap) profiles: which code allocates memory, and how much is still held.
• Allocation profiles: how often memory is allocated, which drives garbage collection overhead.
• Lock and goroutine/thread profiles: where code waits.

Profiles are usually shown as flame graphs: each bar is a function, its width shows how much time it took, and functions it called are stacked on top. Wide bars are where the time goes. You can see at a glance that, for example, 30% of CPU goes into JSON serialisation, or a regex compiled on every request.

Metrics tell you that your service is slow. A flame graph points at the line of code making it slow.

Why continuous profiling

Developers have profiled code for decades, but usually locally, for a few minutes, with special tools. The problem: production behaves differently. Different data, different traffic, different load.

Continuous profiling collects profiles from production all the time, with low overhead, and stores them so you can:

• Look at any point in the past: "what was this service doing during yesterday's spike?"
• Compare time periods or versions: "what changed in CPU usage after release 2.4?"
• See the whole fleet, not just one instance.
• Link profiles to traces, so a slow span shows exactly which code was running.

The tools

• Grafana Pyroscope: open source, integrates with Grafana, supports many languages through SDKs and eBPF.
• Parca: open source, eBPF-based, profiles everything on a node without code changes.
• OpenTelemetry profiling: profiles are becoming the fourth OpenTelemetry signal, with an eBPF-based profiling agent donated to the project. Still maturing, but it's clearly where things are going.
• Commercial: Datadog, Elastic, Google Cloud Profiler, Polar Signals and others.

There are two ways to collect:

• eBPF-based: an agent on each node profiles every process, in any language, with no code changes. Easy to roll out widely. Best support for compiled languages, and growing support for interpreted ones.
• SDK-based: a library in your application. Gives richer information for some runtimes (like memory profiles in Go or Java), at the cost of adding a dependency.

The overhead

The usual first question: does profiling slow down production? Modern continuous profilers sample, for example looking at the stack a hundred times per second, instead of tracing every function call. The overhead is typically small, often around 1% of CPU, depending on the tool and settings.

Compared to the savings it often finds, that's a very good deal. But test it on your workloads, and start with a few services.

How profiling cuts CPU costs

This is where continuous profiling pays for itself. Common findings when teams first look at fleet-wide flame graphs:

• Expensive logging: string formatting and serialisation for debug logs that are never shipped.
• Inefficient serialisation: JSON encoding and decoding dominating CPU in API services.
• Regexes compiled repeatedly instead of once.
• Unnecessary allocations causing high garbage collection overhead.
• Chatty libraries: an HTTP client or ORM doing much more work than expected.
• Hot loops in code nobody looked at in years.

Fixing a single wide bar in the flame graph of a service that runs on hundreds of cores can save real money. That's why profiling is increasingly part of FinOps conversations, not just performance tuning.

A simple way to start

1. Install an eBPF-based profiler (Pyroscope's agent, Parca, or the OTel profiling agent) on a test cluster.
2. Open the flame graph for your most CPU-hungry service.
3. Look for the widest bars that aren't your core business logic.
4. Pick one, fix it, deploy, and compare before and after.

That first comparison, a visibly narrower bar and a lower CPU graph, is usually what convinces the rest of the team.

So, is it worth it?

For any team running services at scale, yes. Continuous profiling fills the gap between "the service is slow" and "here's why", and it regularly finds savings that pay for the whole observability stack.

My take: don't guess where your CPU goes. Look. The flame graph is usually more honest than any of us.

— Desly