← Back to Knowledge
Applied AI · Evidence review

AI software factories optimise the step that was already cheap

AI software factory platforms promise to move a request through to a reviewed pull request, with human approvals at defined points. The question that separates them is where a person signs off: at the end, at the spec, or before any code exists. Independent evidence suggests the problem sits elsewhere.

Where the bottleneck moves

  1. Requestticket · prompt · spec
  2. Generationcommodity · near-free
  3. Pull requestgrowing volume
  4. Reviewthe real bottleneck
  5. Compliancerepository policy
  6. Maintenancewhere cost surfaces

Controlled studies do not match perception

The most rigorous randomised trial measured 16 experienced developers across 246 real tasks in their own mature repositories: with AI assistance they took 19 % longer, while estimating a 20 % improvement. The team later withdrew that result and called the sign unreliable, because between 30 % and 50 % of participants avoided submitting tasks they would have to solve without AI.

  • The retraction matters as much as the figure: there is no single defensible number today.
  • The largest speed-ups reported — 55.8 % and 21 % — come from GitHub and Google measuring their own tooling.
  • The best-designed study, with 151 participants, measured 30.7 % less time while writing and no significant difference once another person had to evolve that code.
  • A two-year study of 26,317 commits found no statistically significant change after adopting an assistant.

Review is the cost nobody publishes

An analysis of 33,707 agent-generated pull requests describes a bimodal regime: 28.3 % merge immediately and the rest join a long queue that is abandoned once subjective feedback appears. Review effort turns out to be predictable before opening the diff, which makes it the central economic variable of the system.

  • Across 22,953 pull requests, less experienced authors produced 4.52 times more review comments and 31 % lower acceptance.
  • Across 23,247 agentic pull requests, the most common description defect is claiming changes that were never implemented.
  • A study of 3.52 million production changes reports increased review effort and 5 % to 8 % higher compute consumption.
  • No study times minutes of review for generated versus written code with the same reviewer and matched tasks.

Declared oversight is not the oversight that happens

This is the finding with the greatest practical consequence. In public repositories, most pull requests co-authored by AI merge with no explicit review, and more than 67 % come from contributors with no prior ownership of that code. That inverts the human pattern, where an outside contributor receives more scrutiny, not less.

  • When review does happen, it is often dominated by other agents; human participation reduces to steering the agent.
  • Review metrics stop being a proxy for human oversight.
  • A benchmark of 823 policies drawn from 12 repositories found agents violating 43.1 % of applicable rules even when the patch is functionally correct.
  • Roughly half of those violations occur in intermediate steps rather than in the final deliverable.

Debt accumulates instead of being resolved

A study of 302,600 AI-attributed commits across 6,299 repositories identified 484,366 issues, 89.3 % of them code smells. More than 15 % of commits from every assistant introduce at least one issue, and 22.7 % of introduced issues were never fixed. The finding is not that the tool produces defects, but that nearly a quarter of them persist.

  • A causal analysis observes front-loaded speed gains that fade, with static-analysis warnings 18 % higher and cognitive complexity 39 % higher.
  • The outcome is not uniform: one agent showed 6.1 % post-merge reverts against 11.5 % for human changes, and another 14.5 %.
  • The meaningful difference lies between agents and between task types, not between “AI” and “people”.
  • The literature proposes separating debt in code from cognitive debt in the team and from missing documented intent.

What remains to be bought is control, not generation

Writing code stopped being the expensive step: the most widely used agent orchestrators are open source with hundreds of thousands of stars. None of the commercial factory platforms appear in the community’s public repositories, because what they sell is the process around the agent. That shift defines what is worth demanding in an evaluation.

  • Which stages require an explicit human decision and which run without one.
  • What happens to downstream approvals when an upstream artefact changes.
  • What share of generated pull requests is discarded, and how much human time review consumes.
  • How repository rules are verified, not merely whether the tests pass.

Official sources