The Verification Tax Nobody Is Measuring

Annie Vella and Kelly Blincoe published a longitudinal study of professional software engineers using AI coding assistants, and it turned up this week in one of the engineering-leadership newsletters I read. The headline number is the one everyone already half-believed: 82% of the developers they surveyed said they were spending less time writing code six months after they’d started using AI assistants regularly. What I think is more interesting, and more useful if you’re trying to lead an engineering org through this, is what happened underneath that number.

Annie Vella and Kelly Blincoe’s New Study

Annie Vella and Kelly Blincoe, both at the University of Auckland, ran two surveys six months apart, in October 2024 and April 2025, asking professional engineers how they were spending their time across six tasks: designing, writing, refactoring, reviewing, testing and debugging. 158 people answered the first survey, and 95 of them answered both, which is the group that matters, because it’s the only one that lets you see how an individual engineer’s answers actually changed rather than just comparing two different crowds.

Writing code dropped the most of the six tasks, by a wide margin. I think the real finding is less that engineers write less code, and more that the balance between creating code and checking it shifted significantly toward checking, in what Annie Vella and Kelly Blincoe call “supervisory engineering work”: directing an AI assistant, evaluating what it produces, and correcting it, rather than typing out a solution yourself. If you’ve spent any time working with an AI coding assistant this year, that framing probably feels obvious. What’s useful is that someone has now measured it properly, with a sample that isn’t just whoever happened to post about it on social media.

The Productivity Perception That Didn’t Move

Here’s the part I think leaders should pay the most attention to. 84% of respondents said their productivity had improved, at both points in time. That sounds like good news, and most dashboards would stop there. But when Annie Vella and Kelly Blincoe looked only at the 95 people who answered both surveys, those productivity ratings barely moved. 77% gave themselves the exact same score six months apart.

Meanwhile, their actual experience was getting worse for a growing number of them. The study measured three dimensions of developer experience: feedback loops, cognitive load and flow state. Feedback loops improved, which makes sense if an assistant is doing some of the waiting-around work for you. But the share of matched engineers who reported their experience had worsened on at least one of those three dimensions nearly doubled, from 14% to 27%, and within that group, flow state was the dominant complaint, rising from 54% to 76%.

There’s one more number in the paper that I think matters more than any of the others. When Annie Vella and Kelly Blincoe asked engineers what concerned them most about AI-assisted development, maintainability was the only concern whose ranking changed significantly between the two surveys, climbing from 3% to 19%. Engineers who are spending less time writing code and more time supervising it are, by their own account, increasingly worried about whether the code will hold up.

Cortex’s Benchmark Numbers Land in the Same Place

Cortex ran a separate benchmark late last year, surveying more than 50 engineering leaders alongside real delivery metrics from multiple organisations, and it makes almost exactly the same argument from a completely different angle. Pull requests per author were up 20% year over year. Incidents per pull request were up 23.5%. Change failure rates were up by roughly 30%. Nearly 90% of the engineering leaders in that survey said their teams were actively using AI tools, but only 45% of their organisations had a formal AI usage policy.

Ganesh Datta, Cortex’s co-founder, described AI as an indiscriminate amplifier: it magnifies whatever practices you already have, good or bad. I think that’s right, and I think it’s the missing half of most AI adoption conversations I’m hearing at the moment. The maintainability worry that Annie Vella and Kelly Blincoe found sitting in a developer’s own head is already showing up as an incident rate in Ganesh Datta’s production data. I think this is one problem, not two, looked at from two different directions.

What Leaders Should Actually Measure

Most of the AI adoption dashboards I’ve seen are built around the numbers that are easiest to get: pull requests merged, lines of code shipped, time to first commit. Those numbers all go up when a team starts using AI assistants properly, which is exactly why they’re popular with anyone who needs to report progress upward. None of them would have caught what Annie Vella and Kelly Blincoe found, because none of them ask what engineers are actually doing with their time, or whether they trust what they’re shipping.

If you’re leading an engineering organisation through this shift, I think the practical move is to start tracking two things you probably aren’t tracking yet: how much of your team’s time is going into supervising and verifying AI output rather than writing it, and whether your maintainability metrics, change failure rate, time to resolve, code churn, are moving in the same direction as your velocity metrics. If they’re not moving together, I think what you actually have isn’t an AI productivity story, but a velocity story with a maintainability bill arriving later, and right now almost nobody’s dashboard is built to see it coming.