What AI changes about software development

What has actually been measured?

Three independent measurement programmes have now published results on AI-assisted development, and they use different methods on different populations. Taken together they describe a consistent picture, which is more than can usually be said in this field.

METR ran a randomised controlled trial in 2025. Sixteen experienced open-source developers worked through 246 real tasks on repositories they had contributed to for an average of five years, with each task randomly assigned to permit or forbid AI tools. Before starting, the developers expected AI to cut completion time by about a quarter. Afterwards they reported a speed-up of roughly a fifth. Measured, they were 19% slower with the tools than without.

The sample is small and the population specific: senior engineers working in code they already know well, using early-2025 tooling. METR has since revised its experimental design. The result should be read as a finding about that setting rather than about all development. What travels beyond the setting is the gap between belief and measurement, which ran in the direction that flatters the tool.

Throughput and instability moved together

DORA’s 2025 report, drawn from a much larger survey population, found AI adoption positively associated with delivery throughput, reversing the previous year’s finding. Instability stayed elevated. DORA’s explanation is volume: code arrives faster than review, testing and deployment can absorb, and the organisations that cope are the ones that already had strong automated testing, mature version control and fast feedback.

That last point deserves attention because it is the practical one. The same tooling produced different outcomes depending on what the receiving organisation already had in place. AI did not create a new capability requirement so much as raise the cost of not having met the old ones.

The artefacts changed shape

GitClear analyses repository history rather than surveying people. Their 2025 report identified 2024 as the first year in which within-commit copy and paste exceeded moved code, the signal they associate with refactoring. Their 2026 analysis, across 623 million changes, reports duplication continuing to rise and refactoring line-moves continuing to fall against a 2022 baseline.

GitClear sells tooling in an adjacent market, and the metrics are its own definitions, so the findings should be read with that in mind. They are still worth taking seriously, because they measure the artefact rather than the author’s impression of the artefact, and because the direction is consistent with what DORA found from an entirely different angle.

Security shows a similar pattern. Veracode tested code generated by more than a hundred models across 80 tasks and found roughly 45% of samples introduced a weakness from the OWASP Top 10, with no systematic improvement from larger models. These are benchmark tasks rather than production repositories, so the number describes model behaviour on a test set. It does not describe the security of any particular codebase.

Measured	Method	Result	What it does not tell us
Task completion time	Randomised trial, 16 senior developers, 246 tasks	19% slower with AI; 20% believed faster	Whether the effect holds for unfamiliar code or newer tooling
Delivery performance	Annual survey at scale	Throughput up, instability elevated	Causation, or which practices carried the difference
Code characteristics	Repository analysis, 623m changes	Duplication up, refactoring down	Whether the duplicated code caused defects
Generated code security	Model benchmark, 80 tasks	~45% of samples carried an OWASP weakness	The security of code after human review

Where did the work go?

Writing code was never the whole job, but it was the visible part, and it is the part that became cheap. What remains has grown in proportion.

Specification comes first. A model produces what it was asked for, quickly and confidently, which raises the cost of asking for the wrong thing. Deciding precisely what a system should do, and what it should refuse to do, is now the constraint on more projects than it used to be.

Review changed in character rather than volume. Reading a change written by someone who cannot explain their reasoning is slower per line than reading a colleague’s work, and there is more of it. Teams that treat generated code as needing less scrutiny because a model is consistent have the relationship backwards.

Verification absorbs most of the difference. Tests, evaluation sets, staged rollouts and fast rollback are what convert an increased rate of change into safe releases, and they are also what DORA finds separating the organisations that benefit from the ones that destabilise.

What we tell clients

Three positions follow from this evidence, and we hold them in engagements.

An organisation that adopts AI-assisted development without changing how it reviews, tests and deploys is buying instability at a discount. The capability arrives immediately; the absorption capacity does not.

Developer self-assessment is not evidence of productivity. Where a client wants to know whether the tooling is working, the answer comes from delivery measurements taken before and after, not from a survey of how the team feels.

Generated code warrants a higher review standard than hand-written code, for the practical reason that the author cannot answer questions about it from memory. In our own work this shows up as smaller changes, more explicit tests around the seams, and a preference for code that can be read quickly over code that was fast to produce.