AI tools are widely perceived to improve developers' speed and efficiency; however, a randomized controlled trial involving 16 experienced developers revealed that those using AI in large software projects took 19% longer to complete tasks. Despite this, AI users were convinced that their productivity surged by 20%. The trial emphasized the importance of contextual evaluations, highlighting risks of blind reliance on AI for complex, interconnected tasks where awareness of overall project implications is crucial.
In large, complex, and important software projects, developers using AI took 19% longer to complete tasks than those not using AI, despite overestimating their productivity.
Evaluations of AI's effectiveness often overlook the context of developers' work, particularly in large projects where tasks are interconnected and require broader awareness.
A failure to appreciate the risks of unexpurgated code entering large projects leads to applying agentic benchmarks inapplicably, making declared results unreliable.
Developers believed AI would improve their efficiency by 24%, yet their actual performance showed a contradiction against this expectation in a controlled trial.
Collection
[
|
...
]