Contrary to popular belief, AI coding assistants may not always enhance developer productivity—especially among experienced coders.
Highlights
- Unexpected Outcome: A randomized study by METR found that seasoned developers completed coding tasks 19% slower on average when using AI assistants like Cursor Pro.
- Initial Expectations vs. Reality: Developers expected a 24% speed boost from AI tools but experienced the opposite—highlighting a clear gap between perception and measurable results.
- Prompting Overhead: Time spent writing prompts, reviewing outputs, and refining suggestions often offset the manual coding time saved by AI.
- Familiarity Matters: While most participants had general AI experience, only 56% had used Cursor before. A steep learning curve likely contributed to slower task completion.
- Low Acceptance Rate: Only 44% of AI-generated code was accepted as-is; the rest required rework or was discarded—adding to the task timeline.
- Contextual Gaps in AI: In complex or unfamiliar codebases, AI struggled to understand nuanced project structure, requiring developers to compensate with more edits and rewrites.
- False Sense of Speed: Despite being slower, developers believed they were 20% faster with AI—suggesting a cognitive bias tied to reduced typing effort.
- Hidden Time Costs: Just 9% of time was spent editing AI code, while over 20% went to prompt creation and managing the AI interaction loop.
- Not a Blanket Verdict: The study focused on experienced developers working in complex environments. Simpler tasks or increased AI familiarity might yield different results.
A new randomized controlled study from the nonprofit research group METR found that AI tools like Cursor Pro can, in certain scenarios, slow developers down rather than speed them up.
Study Overview
The trial involved 16 seasoned open-source developers who completed a total of 246 tasks across familiar repositories. Each participant completed half the tasks using advanced AI assistants such as Cursor Pro and the other half without any AI support.
Developers initially estimated that AI tools would cut completion time by 24%. However, results told a different story—on average, developers took 19% longer to complete their tasks when using AI assistance.
Why AI Slowed Things Down
- Prompting Overhead: Developers spent significant time crafting prompts, waiting for AI responses, and reviewing outputs—offsetting any gains from not typing code manually.
- Tool Familiarity: While 94% of participants had experience with web-based language models, only 56% were familiar with Cursor before the study. Despite receiving onboarding, the learning curve may have contributed to the slower performance.
- Low Code Acceptance Rate: Only 44% of AI-suggested code was ultimately accepted. The remaining suggestions required significant revision or were discarded altogether—adding time rather than saving it.
- Shallow Contextual Understanding: In complex or mature codebases, AI tools often failed to fully grasp the nuances of the project. This forced developers to invest additional time adapting or rewriting code to meet quality and contextual standards.
Perception vs. Reality
Interestingly, developers believed they were more productive with AI, despite the measurable slowdowns. On average, participants perceived themselves to be 20% faster with AI tools—even when they were demonstrably slower.
This perception-reality gap points to cognitive bias, where the presence of AI can create a false sense of acceleration simply by reducing manual effort.
Hidden Costs of Using AI Tools
The study found that only a small portion of time was spent actually editing AI-generated code—just 9%.
A larger portion, over 20%, was devoted to prompt engineering, reviewing suggestions, and managing AI interaction cycles. These hidden time costs often undermined any potential efficiency gains.
Not All Codebases Are Equal
METR emphasized that the findings are not meant to discredit AI tools across the board. The study focused on experienced developers working on complex and unfamiliar codebases—environments where precision, deep context, and high code quality are critical.
In less complex scenarios or for developers with more familiarity using AI tools, results may differ. METR also acknowledged the rapid pace of improvement in AI-assisted development and indicated that newer studies could yield different outcomes in the near future.
