When we rolled assistants out across the engineering organization, the first thing we did was measure the wrong thing. We looked at how much code was being written and how quickly, because that is the metric the tools themselves make visible, and the numbers were genuinely impressive.
Then we looked at how long it took a change to get from a first commit to running safely in production. That number moved far less. In some squads it did not move at all, and in two it got slightly worse before it got better.
What actually got faster
Writing code that a competent engineer already knew how to write. That is not a small category — it covers scaffolding, test setup, the fourth implementation of a pattern we had already agreed on, the shape of an API call in a language someone uses twice a year, and the mechanical half of a large migration.
That work used to consume real hours from people whose scarce contribution was elsewhere. Getting it back is a genuine gain, and I would not give it up.
What did not get faster
Deciding what to build. Agreeing on an interface between two squads. Knowing whether a change is correct in a system no single person holds entirely in their head. Reviewing it. Being confident enough to release it on a Friday.
Generation was never the expensive part of engineering. It was just the visible part.
If your delivery constraint was typing speed, assistants have transformed your organization. Ours was not, and I suspect most are not. Ours was the cost of establishing that a change is right — and we had just made it much cheaper to produce things that needed establishing.
The review queue told us first
Three signals arrived together, and only in hindsight did they obviously belong to the same cause.
Pull requests got bigger. When producing four hundred lines costs what forty used to, people produce four hundred lines. The reviewer's cost per line did not change at all, so total review load rose roughly in proportion to output.
Review latency rose. Predictable, given the above, but it compounds badly: slower review means more parallel branches in flight, more conflicts, more rebasing, and more context lost between writing a change and defending it.
Approvals got faster per review. This was the one that worried me. Reviewers were spending less time on more code, and the reason was that the code read well. It was conventional, consistently named, plausibly structured. It looked like something a careful engineer had written.
Plausible is a new failure mode
Hand-written wrong code usually looks wrong. There is an awkward variable name, a comment that does not match the branch below it, a function that clearly grew in two sittings. Reviewers have spent their careers learning to read that texture, and it is a large part of what makes review work at all.
Generated wrong code does not have that texture. The mistakes are not sloppy, they are confident: the third case nobody thought about, a retry that silently swallows the error class you actually needed to see, an off-by-one in pagination that only appears past a page boundary the test data never reaches, a currency rounding decision made silently and consistently in the wrong direction.
None of that is a new category of bug. What is new is that the usual surface cues no longer correlate with the presence of bugs — and reviewers are calibrated on those cues whether or not they know it.
The question we ask in review now
Not "does this look right" but "can the author explain why this is correct without re-reading it?" An author who has to scroll back through their own change to answer has not yet understood it, and a reviewer is not going to understand it better in less time.
What we changed
- We got stricter about size, not looser. The instinct was to relax review standards because output had gone up. The correct move was the opposite: a generated eight-hundred-line change is not cheaper to review than a hand-written one, and treating it as though it were is how the whole benefit leaks back out as defects.
- The author owns understanding. "That is what it produced" is not an answer in a review, in an incident, or in a design discussion. Whoever puts their name on the pull request is accountable for every line in it in exactly the way they always were. Stating this explicitly mattered more than I expected.
- Tests became the artifact we scrutinise hardest. Generated tests have a specific and dangerous habit: they assert what the implementation does rather than what the requirement is. A suite like that passes forever and protects nothing. We now review the test before the implementation, which also turns out to be a good habit independent of any of this.
- We pointed it at the reversible work first. Internal tooling, migrations with a mechanical shape, test coverage for legacy code that had none, throwaway scripts. High volume, low consequence, fast feedback. That is where the leverage is real and the downside is bounded.
The part I am still working out
Junior engineers.
Judgment is formed by struggling with a problem slightly beyond you and eventually resolving it. That struggle is exactly what an assistant is best at removing. I do not think the answer is to withhold the tools — that produces engineers who are worse at the job they will actually have — but I am fairly sure the answer is not to change nothing either.
What we have landed on for now is changing what we ask of people early on. More reading of existing systems, more pairing on debugging where the fault is genuinely unclear, more explaining a change to someone else before it is merged. The through-line is that we are asking for demonstrated understanding rather than demonstrated output, because output has stopped being evidence of understanding.
Ask me in a year whether it worked. I am not going to pretend I know yet.
Where it is unambiguously good
I want to be clear that the honest accounting still comes out positive, and in some places strongly so.
Navigating an unfamiliar part of a large estate, where the previous option was interrupting the one person who remembers. Retrofitting tests onto code written before anyone was writing them. The mechanical eighty percent of a framework migration. One-off tooling that was never worth a ticket and therefore never existed. Getting from a blank file to something concrete enough to argue about.
That last one is underrated. A great deal of engineering time goes into converting a vague intention into a specific proposal that a team can react to, and lowering the cost of a first draft lowers the cost of being wrong in public — which is a cultural improvement as much as a technical one.
What I would tell another organization
Find out where your delivery actually stops before you measure whether the tools helped. If changes sit waiting for review, for a decision, for an environment, or for someone's confidence, then generating code faster fills the queue in front of the constraint rather than relieving it — and a longer queue in front of a bottleneck looks exactly like progress right up until it does not.
The organizations getting the most out of this are not the ones producing the most code. They are the ones that noticed the constraint had moved and rebuilt review, testing and ownership around where it moved to.