A reviewer sees an AI-generated answer that looks fluent and clicks approve without checking the source document. The answer contains a subtle error that the reviewer could have detected. AI overreliance describes a pattern in which people give an AI output more decision weight than its accuracy or scope warrants, especially when they follow a wrong recommendation despite available contrary evidence. It is a behavioral risk, not a claim that every use of AI is careless. The same reviewer may appropriately rely on the tool in other cases.
What counts as overreliance
Overreliance needs a criterion. A person who accepts an AI recommendation that turns out correct has not necessarily overrelied, even if they checked little; the decision process may still be fragile. A person who rejects a correct recommendation may underuse a useful tool. Researchers can study both by presenting cases where the AI is correct and incorrect and observing whether people distinguish them. The most informative measure is conditional: what happens when the human and AI disagree, and which one has better evidence for that case?
Automation researchers described misuse and disuse before current generative AI systems. Parasuraman and Riley analyzed how people can rely on automation inappropriately or fail to use it when useful. Skitka, Mosier, and Burdick studied automation bias in decision tasks, including commission errors when people follow an incorrect recommendation and omission errors when they miss an event because an aid does not flag it. These distinctions remain relevant, though a language model’s open-ended text differs from a cockpit alert or a binary decision aid.
Overreliance is related to trust calibration but narrower. Trust calibration asks whether expectations and reliance match capability across cases. Overreliance focuses on excessive reliance, particularly harmful acceptance of wrong outputs. A user can express skepticism and still overrely under deadline pressure. Another can trust a tool strongly while checking the right cases and achieving appropriate reliance. Measuring stated trust alone cannot diagnose the behavior.
It also differs from automation bias as a historical research category. Automation bias includes specific omission and commission errors under automated aid conditions. AI overreliance may include copying fabricated citations, accepting flawed summaries, or failing to inspect generated code. The mechanism may involve fluency, workload, social pressure, or misplaced assumptions about model capability. Calling every such error “automation bias” without studying the context can obscure the route to prevention.
Why fluent output can be persuasive
Errors in generated text can be especially hard to catch when they are woven into otherwise accurate material. A citation may name a real author but attribute the wrong finding; a summary may preserve the main point while reversing one condition. Checking only the surface form or one familiar sentence will miss these failures. Verification should target consequential claims and boundary conditions, not merely whether the output reads smoothly. The more persuasive the answer, the more useful a source-linked review process can be.
Generative systems often produce well-structured explanations even when a detail is wrong. Form and confidence can create an impression of authority. A user who is unfamiliar with the topic may struggle to separate a plausible explanation from a true one. But fluency is not the only cause. Interfaces may hide source evidence, make copying easy, or place the answer before the primary material. Organizational targets may reward speed over verification. The behavior emerges from a system of design and incentives, not only from an individual’s attitude.
Explanations can sometimes increase reliance on a wrong answer. Bansal and colleagues found that AI explanations did not automatically create complementary human-AI team performance in their tested tasks. The lesson is not that explanations are useless. It is that an explanation must be evaluated for whether it helps users discriminate correct from incorrect advice. A rationale that merely sounds coherent may be especially dangerous if it encourages acceptance without exposing a verifiable basis.
Authority cues matter as well. A model output embedded in an official workflow can appear vetted, even when it is a draft. A confidence score can look precise while being poorly calibrated for the current population. A badge saying “AI verified” may be interpreted as proof of accuracy without showing the validation behind it. Designers should test what users infer from these cues. Misleading assurance can generate overreliance even if the underlying model has not changed.
Repetition can change habits. If an AI system is usually right on easy cases, users may reduce scrutiny and carry that habit into difficult cases where errors are more frequent. This is a monitoring problem as well as a trust problem. A person may not receive feedback when an error occurs, so their impression of reliability stays optimistic. Evaluations should include rare but consequential failures and examine behavior over time rather than after one introduction to the tool.
Evidence from human-AI studies
Automation-bias research has long distinguished commission from omission. A commission error follows an incorrect directive despite contrary information. An omission error can occur when a person fails to act because the automated aid did not flag an issue. A generative AI interface can produce both: users may accept a false claim and may overlook a missing fact because the summary did not mention it. Measuring only false statements in the output misses the second route. Reviewers need access to the original material to notice absence as well as error.
Buçinca and colleagues tested cognitive forcing functions designed to make users engage more thoughtfully with AI advice. Their experiments found reductions in overreliance in the studied tasks, with tradeoffs in effort and experience. A design that slows people down can help when the task justifies checking, but it can also burden routine decisions. The result supports testing friction selectively, not placing a universal obstacle in every interaction.
Comparing human-alone, AI-alone, and combined performance can reveal whether the team adds value. If the combined team is worse than the better component alone, overreliance may be one contributor. Yet team performance depends on case mix, training, and interface. Researchers should report where the human was initially right, where the AI was right, and how advice changed the final answer. Aggregate accuracy can hide the exact failure mode.
A study should also ask whether the human could realistically detect the error. If the source material is unavailable, the AI’s wrong answer cannot be checked in the ordinary workflow. Labeling acceptance “overreliance” may describe the outcome but misplace blame. The organization should provide evidence, time, and escalation routes. When ground truth is only known to researchers after the task, users may be making reasonable decisions under limited information. The design question is how to change those conditions.
Expertise can reduce some errors but does not eliminate overreliance. An expert may notice a fabricated citation while overlooking a familiar-looking summary error under time pressure. A novice may be more alert to an obviously strange output but lack domain knowledge for subtle cases. Studies need representative users and tasks. A quick online experiment with general participants cannot establish how a trained professional will behave after months of deployment.
Practical checks that preserve human judgment
Checklists can be useful when they name concrete failure modes: Are numbers copied accurately? Does each citation support the stated claim? Are exceptions and eligibility conditions preserved? A generic instruction to “verify everything” can be impossible under real workloads and may become a ritual. Risk-based review prioritizes claims that could change a decision or harm someone. The checklist itself should be tested, because extra steps can create fatigue and shift attention away from an unlisted failure.
Interface design can make correction socially easier. If challenging the AI requires a long explanation while accepting it takes one click, the workflow nudges users toward reliance. A neutral “needs verification” state and simple access to sources can reduce that asymmetry. The organization should record when users disagree with the tool and treat those cases as learning opportunities rather than signs of obstruction. This is a design hypothesis to evaluate, not a guarantee that a button alone changes behavior.
A useful workflow can ask the person to inspect primary evidence before viewing the AI’s conclusion on selected high-stakes cases. It can display source passages next to claims, make uncertainty visible, and require a reason when overriding contrary evidence. These are candidate design interventions, not guaranteed fixes. They should be evaluated for whether they reduce wrong acceptance without causing excessive delay or rejection of correct help.
Verification must fit the task. For a factual summary, the reviewer can compare each consequential claim with the source. For code, tests and review can examine behavior beyond the generated explanation. For medical or legal decisions, qualified judgment and formal procedures remain essential. A generic banner saying “AI can make mistakes” gives little guidance about which parts to check. Specific failure-mode information is more actionable.
Separation of duties can help. The person who produces an AI draft need not be the only person who approves it for publication or use. Independent review can catch errors that the original user has become anchored to. But adding a second person is not sufficient if both see the same persuasive answer first or lack source access. Review quality depends on evidence and authority, not only on headcount.
Monitoring after deployment is necessary because model performance and tasks change. A system may face new document formats, policy revisions, or populations. Users cannot calibrate to drift they cannot see. Organizations can audit sampled decisions, track corrections, and provide feedback about known failure modes. This supports learning and accountability while avoiding the fiction that a one-time training session permanently solves overreliance.
Ethical and organizational stakes
Training should cover the system’s actual error profile, including examples of plausible wrong answers. A lecture about general AI risk may be less useful than practice identifying the tool’s common failure modes on representative cases. Feedback should be refreshed as the model or task changes. Workers also need permission to spend time checking and to pause a decision when evidence is insufficient. Without that support, training can become a formal excuse to leave foreseeable risks with the frontline user.
The final human click does not make an error solely the human’s responsibility. Developers, deployers, and managers decide how the recommendation is presented, whether verification is possible, and what speed is expected. An organization that removes time for checking while requiring human sign-off has created a predictable risk. Responsible governance includes model validation, meaningful human authority, documentation, and a route to challenge the output.
Overreliance can harm people unevenly. A model may fail more often on underrepresented cases while looking accurate overall. If users are shown only aggregate performance, they may follow wrong advice precisely where independent scrutiny is most needed. Audits should examine conditional performance and downstream effects. A claim that a system “beats humans” on average is not enough to justify unqualified reliance in every subgroup or situation.
Underreliance matters too. Rejecting useful AI assistance can waste time or miss a correct warning. A balanced evaluation records both wrongful acceptance and wrongful rejection. The aim is not to make people wary of every output, but to make reliance discriminating. Good human-AI collaboration allows the person to use strong evidence from the tool while recognizing when its output is unsupported.
What the evidence does not show
Nor does it show that simply hiding AI output until the last step is always best. In some tasks early advice may reveal information the person would otherwise miss. The design choice depends on the tool’s error profile and the user’s ability to evaluate evidence. Comparing workflows on correct and incorrect AI cases gives a better answer than assuming any one sequence is universally safe.
Research does not show that every AI-assisted error is caused by individual laziness, that explanations always prevent overreliance, or that one prompt telling users to “think critically” solves the problem. Laboratory interventions can reduce errors under specified conditions but may impose costs or fail in other domains. High average AI accuracy does not establish appropriate reliance on rare cases. A full account needs the task, model failure mode, available evidence, user behavior, and institutional constraints.
The reviewer who approved a fluent but wrong answer needed more than a warning to be careful. They needed a way to inspect the claim, time to do it, and a workflow that rewarded correction. AI overreliance becomes a useful concept when it points to those observable conditions and supports tests of whether a changed system leads to better decisions.
Research references
- Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39, 230–253. View source
- Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51, 991–1006. View source
- Skitka, L. J., Mosier, K. L., & Burdick, M. D. (2000). Accountability and automation bias. International Journal of Human-Computer Studies, 52, 701–717. View source
- Bansal, G., et al. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. Proceedings of CHI 2021. View source
- Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5, 1–21. View source