The use of AI coding assistants would slow down the work of experienced software developers by about nineteen percent, according to a study by METR.
Artificial intelligence is often seen as a tool to speed up tasks. However, this does not seem to be the case for software developers. According to a study by the American organization Model Evaluation & Threat Research (METR), software developers work nineteen percent slower with AI than without.
A different study by Uplevel showed similar results, namely that AI coding assistants did not lead to increased productivity, but rather resulted in 41 percent more bugs in the code.
Slower with AI
The research was conducted with sixteen experienced developers from large open-source repositories to which they had contributed for several years. “Surprisingly, we see that when developers use AI tools, they take 19 percent longer than without: AI makes them slower”, said a researcher from METR. “This is a significant delay that contradicts the beliefs of developers and the predictions of experts.”
Another striking finding from the research is the gap between perception and reality. The developers expected AI to speed them up by 24 percent. Even when they experienced delays, they still believed AI had sped them up by twenty percent.
Determining factors
According to the research, the delay can be attributed to various factors. There is an “exaggerated optimism about the usefulness of AI”, leading developers to have unrealistic expectations. Furthermore, the participants were so experienced that AI assistance might have had nothing to offer them. Another possible factor relates to the low reliability of AI. The developers accepted less than 44 percent of the generated suggestions but spent a lot of time cleaning up and reviewing them.
read also
Smarter Coding with the Codex AI Agent in ChatGPT
“The delay we observe does not mean that current AI tools do not often improve developer productivity – we find evidence that the developers’ high familiarity with repositories and the size and maturity of the repositories both contribute to the observed delay, and these factors do not apply in many software development environments”, the researchers conclude.