How to Measure If an AI Coding Agent Saves Time

The most useful result in this area is uncomfortable. In a randomised controlled trial run by METR, experienced open-source developers working on repositories they knew well took 19 percent more time to complete coding tasks when allowed to use AI tools, and afterwards estimated that AI had made them 20 percent faster.

That is a 39-point gap between perception and measurement, in the direction that flatters the tool. It does not prove AI coding agents are useless. It proves that asking the people using an AI coding agent whether it saves time is not a measurement, and that includes asking yourself.

So measure. Four numbers are enough, and none require new tooling.

The two numbers that mislead

Worth naming first, because they are the ones dashboards default to.

Lines of code and commit count. An agent trivially inflates both. More generated code with the same functionality is a cost, paid later, by whoever maintains it.

Pull requests merged. Better, and still gameable in the direction that matters. An agent that produces three small PRs where a developer would have produced one coherent one looks like a 200 percent improvement and is not. Worse, PR count says nothing about whether the change was correct, which is where the agent's real risk lives.

Both share a flaw: they measure output, and the thing you actually care about is how long it took to get to working, reviewed, shipped software that did not come back.

The four numbers that hold up

1. Cycle time from first commit to merge

Wall-clock hours from the first commit on a branch to the merge. Not developer-hours, not story points. It is already in your git history, which means you can measure the last three months retroactively before you change anything.

This is the headline number, and it captures what raw speed of code generation does not: review, rework, and the time a PR sits waiting because it was hard to understand.

2. Review rounds per pull request

Count the distinct times a reviewer requested changes before approval. AI-generated PRs frequently need more rounds, because the code works and is structured in a way the team would not have chosen, or contains a subtle deviation nobody catches on the first pass.

If cycle time drops but review rounds climb, you have moved work from the author to the reviewer. That can still be a win if the reviewer is faster than the author was, and it is a loss if you have one senior reviewer who is now the bottleneck for everybody.

3. Rework rate at seven days

The share of merged PRs that get materially modified within a week, excluding planned follow-ups. Approximate it by looking for changes to the same files by the same author shortly after a merge, then confirm by hand on a sample. It is a rough metric and still the most informative one, because it is where speed that was not real shows up.

A rising rework rate alongside falling cycle time is the classic pattern of an agent shipping plausible code fast. The time was not saved, it was borrowed from next week.

4. Unattended share

What proportion of the agent's work went in with the developer doing something else rather than watching. This is the number that separates real leverage from fast autocomplete. If a developer is supervising every token, you have at best sped up typing. If a task genuinely runs while they review something else, you have added capacity.

Track it crudely: for each agent task, was the developer able to do other work during it? A weekly tally by hand for a month is enough.

How to run the comparison

Four weeks, no new tools, roughly two hours of setup.

1. Pull four weeks of history as your baseline before changing anything. Cycle time and review rounds are both derivable from your git and PR data. Note the median and the spread, not just the average, because a few very slow PRs will otherwise dominate.

2. Categorise tasks by type. Bug fix, feature, refactor, test coverage. Agents perform very differently across these, and a single blended number will hide it. Most teams find the wins concentrated in a narrow band.

3. Run four weeks with the agent, changing nothing else. No new process, no new framework, no reorganisation. One variable.

4. Compare within categories, then decide per category rather than overall. The realistic outcome is not "it works" or "it does not", it is "it is clearly worth it for test coverage and boilerplate, neutral for features, and negative for anything touching the payments module".

Two things that will corrupt the result if you let them. Do not let people choose which tasks go to the agent, because they will choose the ones it is good at and the comparison becomes meaningless. And do not compare a week containing a release to a week that does not.

Interpreting what you get

The honest read on most first measurements: a modest improvement in a few categories, no change in most, and one category where it is clearly worse. That is a useful result, and it is more actionable than a headline percentage.

Watch for the specific pattern where the agent helps most on tasks in unfamiliar code and least on tasks in code the developer wrote last month. That is consistent with the METR finding, where the developers were working in repositories they knew deeply, and it suggests routing rather than a blanket adoption decision.

If you want to compare specific agents against each other rather than agent versus no agent, that is a different exercise with a different method, covered in how to benchmark AI coding agents on your codebase. And if review rounds turn out to be your bottleneck, the fix is usually the review process rather than the agent, addressed in how to review AI-generated code before you ship it.

Turning hours into a decision

Once you have hours, the money question is simple arithmetic, and it is worth doing before renewing an annual plan. Multiply hours saved per developer per month by a loaded hourly cost, subtract subscription and token spend, and subtract an honest estimate of the review overhead you added.

Teams frequently discover the tool pays for itself several times over on cost while the bottleneck they actually care about, which is usually senior review capacity, got worse. Both facts can be true, and only one of them shows up in the invoice. The general framing for that calculation is in how to measure AI ROI for a small business, and for scoping a project before you start, how to estimate AI project ROI before you start.

For choosing which agent to adopt in the first place, AI coding tools covers the selection criteria that matter before you get to measurement.

Frequently asked questions

Is four weeks long enough?

For a team shipping several PRs a week, yes, with the caveat that the first week includes a learning curve. If your volume is low, extend rather than accepting a noisy result from a handful of PRs.

Should I tell the team they are being measured?

Yes. Covert measurement of developers is corrosive, and the metrics here are team-level rather than individual. Frame it as evaluating the tool, which is what it is, and make sure nobody's performance review touches these numbers or the numbers stop meaning anything.

What if the agent makes work more enjoyable but not faster?

That is a legitimate reason to keep it, and worth stating plainly rather than dressing up as a productivity claim. Retention and morale are real. They are just a different argument from time saved.

Does the METR result mean AI coding tools do not work?

No. It measured experienced developers in codebases they knew well, using early-2025 tooling, which is close to the least favourable case. The useful conclusion is that self-reported speedup is unreliable, so measure your own situation rather than assuming either direction.