Some developers say AI has dramatically improved their productivity, while others argue that it mostly generates low-quality code that takes substantial human effort to fix. Are there studies or surveys examining why experiences differ? Possible factors might include the programming domain, language, project size, tooling or model quality, developer experience, workflow, ability to provide context, attitudes toward AI, and how productivity is measured.
5 Answers
People also measure different things. Someone may call AI a huge success because it produces code quickly, while someone else includes review time, debugging, maintenance, regressions, and long-term code quality. The strongest results usually come when the developer supplies a solid plan, uses tests and linters, reviews the output, and treats the model as an assistant. Letting it generate large amounts of unverified code can create an impressive-looking but expensive mess.
The biggest variables seem to be developer experience, project size, and how well-defined the work is. An experienced developer who understands the architecture can use AI as a force multiplier, while someone without enough context may accept plausible-looking mistakes. AI is especially useful for repetitive tasks, prototypes, tests, scripts, and exploring alternatives, but it is much less reliable when requirements are unclear or the codebase has complicated hidden constraints.
There is at least some empirical evidence, but it is still limited and results depend heavily on the setup. One widely discussed study involving experienced developers working in familiar, mature codebases found that participants expected AI to make them faster, while the measured completion time actually increased. That does not prove AI is always harmful; it shows that perceived productivity and actual delivery speed can diverge, especially when reviewing and correcting generated code is included.
Small studies should not be treated as universal answers, though. Model quality, tools, task selection, and the date of the experiment can change the result substantially.
The domain makes a huge difference. AI often does well with common languages, conventional web code, boilerplate, shell scripts, and well-tested components. It is much riskier with unfamiliar frameworks, low-level systems, firmware, security-sensitive logic, event ordering, or codebases whose important rules are undocumented. A task that looks simple to the model can hide assumptions that only a domain expert would notice.
Using AI effectively is a skill. It takes practice to learn what context to provide, how to break work into manageable steps, and which tasks should never be delegated without close review. I get the most value from using it as a patient design partner, debugging assistant, and prototyping tool rather than as an autonomous software factory. The person still needs to understand the requirements, architecture, and generated code.
Organizational maturity matters just as much. If requirements keep changing or stakeholders cannot agree on what they want, developers may spend most of their time in meetings and rework. AI cannot solve that bottleneck.