When combinations of humans and AI are useful - A systematic review and meta-analysis
- Author
- Michelle Vaccaro, Abdullah Almaatouq, Thomas Malone
- Published
Abstract
Inspired by the increasing use of artificial intelligence (AI) to augment humans, researchers have studied human–AI systems involving different tasks, systems and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes. We searched an interdisciplinary set of databases (the Association for Computing Machinery Digital Library, the Web of Science and the Association for Information Systems eLibrary) for studies published between 1 January 2020 and 30 June 2023. Each study was required to include an original human-participants experiment that evaluated the performance of humans alone, AI alone and human–AI combinations. First, we found that, on average, human–AI combinations performed significantly worse than the best of humans or AI alone (Hedges’ g = −0.23; 95% confidence interval, −0.39 to −0.07). Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when AI outperformed humans alone, we found losses. Limitations of the evidence assessed here include possible publication bias and variations in the study designs analysed. Overall, these findings highlight the heterogeneity of the effects of human–AI collaboration and point to promising avenues for improving human–AI systems.
Main
When do humans and AI complement each other, and by how much?
A large body of work suggests that integrating human creativity, intuition and contextual understanding with AI’s speed, scalability and analytical power can lead to innovative solutions and improved decision-making in areas such as health care1, customer service2 and scientific research. However, a growing number of studies reveal that human–AI systems do not necessarily achieve better results than the best of humans or AI alone. …
These seemingly contradictory results raise important questions: when do humans and AI complement each other, and by how much?
Results
Human-AI synergy: Human-AI system vs. max(human, AI)
We found that the human–AI systems performed significantly worse overall than this baseline. The overall pooled effect was negative (g = −0.23; t92 = −2.89; two-tailed P = 0.005; 95% confidence interval (CI), −0.39 to −0.07) and considered small according to conventional interpretations.
Human augmentation: Human-AI system vs. human alone
The human–AI systems performed significantly better than humans alone, and this pooled effect size was positive (g = 0.64; t98 = 11.87; two-tailed P = 0.000; 95% CI, 0.53 to 0.74) and medium to large.
Discussion
Interesting points:
- Adding AI to tasks humans do better than AI produces better results than humans working alone.
- Adding humans to tasks AI does better than humans produces worse results than AI working alone.
The authors speculate that only when humans are generally better than AI can they also judge better whether to accept AI’s suggestions. In other words, when humans perform worse than AI, they fail to question it when they should (overreliance), or insist on their own judgment when they should accept its suggestions (underreliance).
Another interesting point is that, whether humans were better or worse than AI at the task, working with AI was generally better than working alone. This was especially true for open-ended tasks (such as content creation), rather than tasks involving a choice among a finite set of options.