A clinician reviews a scan while an AI tool highlights one diagnosis. A public employee reviews an application beside a risk score. A pilot watches an alert system decide which signal to display. In each setting, the person remains responsible for noticing what the tool misses and refusing advice that does not fit the case. The difficulty is that a recommendation can change attention before the final decision is made.
Automation bias is a family of problems in which people rely on automated output in ways that produce avoidable errors, including following a wrong recommendation or failing to look for an unflagged problem. It is not a synonym for all trust in a tool. If a system is accurate, taking its advice can be sensible. The evidence review asks when reliance becomes inappropriate, how it has been measured in different domains and what organizational accountability requires when the human and system disagree.
What the classic experiments established
Skitka, Mosier and Burdick tested decision-making with an automated aid in simulated tasks. Participants sometimes committed errors of commission, following incorrect recommendations despite other valid information, and errors of omission, missing events the aid failed to flag. The distinction remains useful. A person can be misled by a false alert or by the absence of one. A study that measures only agreement with explicit advice misses the second pathway.
The experiments did not prove that all professionals will behave identically, or that every AI interface will create the same effect. They showed a failure mode under controlled conditions, where the aid’s performance and the available independent cues could be varied. That made “human in the loop” an empirical question: did the participant use the other evidence, and what did they do when automation was wrong?
Goddard, Roudsari and Wyatt later reviewed automation-bias studies with a healthcare focus. Their systematic review examined frequency, mediating factors and possible mitigations across a mixed evidence base. The literature did not yield one reliable percentage of error for every clinical setting. Tasks, aid accuracy, user expertise, workload and outcome definitions varied. The review helped establish why evidence from a laboratory display must be translated cautiously before it supports a patient-safety claim.
Correct advice and wrong advice require separate counts
Suppose an AI tool recommends the correct diagnosis in 90 of 100 cases. A clinician who agrees with it 90 times might be appropriately using it. Another clinician who agrees with it 100 times may still look excellent on a simple agreement dashboard, but the ten wrong cases reveal overreliance. The evaluator needs an independently justified answer for each case and should report acceptance of correct advice and rejection of wrong advice separately.
Baseline human performance is also important. An aid may improve accuracy overall while causing a specific kind of new error. Wang and colleagues tested AI-supported interpretation of knee MRI cases with clinicians and found that support could improve performance, while incorrect AI recommendations created automation-bias risk. Their study explored suppressing AI output in some conditions as a mitigation. The result is bounded to its task, participants and design; it does not establish that suppressing recommendations is the best policy in every clinic.
For a responsible comparison, ask what the clinician would have decided without AI, what evidence was available, whether advice was right and what happened after an incorrect suggestion. A changed answer can be a benefit or a harm. Counting changes without their direction collapses the central distinction. The same principle applies outside medicine, although the definition of a correct answer can be harder when policy, values or uncertainty matter.
Public-sector advice has more than one bias pathway
Alon-Barkat and Busuioc studied human–AI interactions in public-sector decision making, including a study with Dutch civil servants. Their experiments distinguished automation bias from selective adherence: a decision maker may follow advice more readily when it fits an existing stereotype or expectation. Their first two studies found selective-adherence patterns with both human and algorithmic advice, without a stronger pattern for the algorithmic source; the civil-servant study did not yield the same result. That nuance matters. Not every unfair decision involving a tool is blind deference to it.
Selten, Robeer and Grimmelikhuijsen experimentally studied Dutch police officers using a mock predictive-policing aid. The officers tended to trust recommendations that confirmed their professional judgment. Explanations did not show the hoped-for effect on trust in that design. The study suggests that professional experience can help reject implausible advice, yet it can also preserve an existing bias when a recommendation appears to validate it. Expertise and discretion should be investigated, not assumed to be either a perfect safeguard or an obstacle.
These studies use simulations or mock systems, not direct observation of every consequential public decision. Still, they challenge a simple story in which AI bias comes only from a biased model and human oversight automatically corrects it. Human expectations, task instructions, institutional incentives and the presentation of advice can change the combined outcome. A fairer model may not improve decisions if it is selectively ignored; a flawed model may be especially harmful when its output fits an existing expectation.
A Swedish public-service case tests institutional trustworthiness
Berman, de Fine Licht and Carlsson examined a decision-support application in the Swedish Public Employment Service. They reported gaps between aspirations for trustworthy AI and features of the deployment, including transparency, intelligibility and stakeholder involvement. Their case study offers Nordic public-sector evidence about implementation conditions. It is not a randomized estimate of automation-bias frequency, and it should not be used to claim that every Swedish agency has the same problem.
The case is valuable because an interface is embedded in an institution. A worker’s ability to challenge a score depends on training, documentation, time and the authority to escalate. A person affected by a decision may need an explanation and a route to contest it. If those processes are weak, the presence of a named human reviewer may not create meaningful accountability. The study directs attention to what the service actually gives staff and citizens, not just what an AI policy promises.
Public-sector decisions also have multiple legitimate aims. Accuracy may matter, but so do procedural fairness, equal treatment, legal obligations and the person’s opportunity to be heard. An algorithm’s statistical target may capture only one part of that task. Professional judgment can provide context, yet it too is fallible. The evaluation should identify which values the system is intended to support and how disagreement is resolved when they conflict.
High-risk systems need a workable oversight path
Article 14 of the EU AI Act addresses human oversight for high-risk AI systems. Its text calls for capabilities such as understanding a system’s limitations, monitoring operation, awareness of possible overreliance, interpreting outputs, disregarding or reversing them and intervening or stopping operation where appropriate and proportionate. The legal text specifies obligations in its scope. It does not demonstrate that any particular deployment meets them or that a user will exercise them well under pressure.
A workable oversight path begins before the wrong recommendation appears. The operator needs enough knowledge of the task and the system’s likely failures. The interface must display relevant evidence and make disagreement feasible in the available time. The organization must define what happens after a challenge: who reviews it, who can suspend a faulty tool, and how people affected by earlier decisions can obtain correction.
Training can improve recognition of errors, but training alone is not a substitute for authority. A worker who correctly spots an anomaly yet cannot reverse the output has not been given effective oversight. Nor does an “override” button solve the problem if using it carries an unexplained penalty, consumes too much time or requires a justification that cannot be written from the information available. The relevant test is what the person can do in the real workflow.
Do not confuse automation bias with rejecting useful advice
At the other extreme, a person may reject useful algorithmic advice too readily. It is a reminder that “less reliance” is not automatically safer. An oversight program should aim for appropriate reliance: use sound advice when it helps and reject misleading advice when independent evidence warrants it.
Appropriate reliance is difficult to measure when a decision has no immediate ground truth. A public-service decision may combine prediction with rules and individual circumstances. An auditor can still construct cases with known factual errors or clear procedural violations, then evaluate how reviewers respond. For ambiguous cases, the record should include reasons, uncertainty and the perspectives of people affected, rather than forcing every disagreement into a binary correct/incorrect label.
A dashboard of override rates alone cannot resolve the question. A high rate could reflect poor AI, conscientious review or systematic human rejection of valuable advice. A low rate could reflect good AI or deference. The evaluator needs case mix, model performance, baseline human judgments, task stakes and the practical cost of dissent. Claims about automation bias should be tied to those conditions.
Accountability spans design and deployment
When a professional makes the final decision, it is tempting to place full responsibility on that person. Yet model objectives, training data, procurement, display design, performance targets and staffing are often set elsewhere. The professional may bear a duty to review, but the institution controls much of the environment in which review occurs. Accountability should follow the actual distribution of power and information.
The Swedish case study and public-sector experiments point to complementary questions. Did the organization choose a defensible task for AI assistance? Did it validate performance for the people and situations in which it would be used? Did it tell workers what the score means and what it does not? Did it make challenge and appeal possible? Did it examine whether different groups were harmed or helped? A post-hoc search for a mistaken click will miss failures built into the workflow.
A useful audit trail records the model version, the recommendation, the evidence displayed, the human action and the reason for a disagreement where appropriate. It should support correction, not merely blame. Privacy and labor concerns must shape how logs are used. A system that records every worker’s hesitation for punishment could discourage careful review and thereby undermine the very oversight it claims to measure.
Evaluate mitigations against the failure they target
Possible mitigations include independent first judgments, second looks at unflagged cases, selective hiding of AI advice, clearer uncertainty displays, time for review and escalation rules. Each has a cost and a failure mode. Hiding advice may reduce contamination by a wrong recommendation but also withhold a useful signal. Adding steps may improve deliberation in a trial but create unsafe delay in an urgent setting.
Wang and colleagues’ clinical study is useful precisely because it examined a specific strategy in a specified task rather than declaring one interface fix universal. A public-sector pilot should likewise compare actual decision quality and burden under alternative designs. It should ask whether benefits persist when operators are busy, cases are atypical and the AI’s error rate changes. If a mitigation only works in a quiet demonstration, its operational value remains uncertain.
Involve affected people in the evaluation. A worker can explain where advice is confusing or overrides are costly; a patient or citizen can explain whether a decision and appeal process is intelligible. These accounts do not replace quantitative outcomes. They help identify mechanisms and harms that a narrow accuracy metric could overlook. A correction process should allow the organization to change the tool, not only teach users to cope with it.
What the evidence does not show
The classic and clinical studies do not provide a single universal automation-bias rate. Their tasks, participants and aids differ. Goddard and colleagues’ review summarizes a heterogeneous literature, while Wang and colleagues’ MRI simulation does not establish patient outcomes in routine care. A reduction in one error type may come with effort or missed benefit elsewhere.
The public-sector experiments do not show that officials always trust AI blindly. They identify selective adherence and context-dependent use of advice. The Swedish case study documents implementation gaps in one service; it does not quantify all Nordic public systems. Article 14 states requirements for high-risk systems, not empirical evidence that those requirements have been met.
Nor does this review settle the right balance between machine and human judgment for every decision. The Institute synthesis is to evaluate the combined system: correct and incorrect advice, omissions and commissions, professional reasons, institutional incentives, appeal and the distribution of consequences. A final human signature is not proof of safety, and a model error is not the whole explanation of a harmful outcome.
Return to the three workstations
The clinician needs a reason to trust or reject a highlighted diagnosis. The public employee needs to know what a risk score leaves out and how to hear a person’s circumstances. The pilot needs to notice the event that an alert failed to display. In all three cases, human review is a real task, not a decorative last step.
Automation bias is best understood as a testable risk in the relation between people and aids. The point of oversight is not to demand perfect skepticism from a worker. It is to build conditions in which useful advice can be used, wrong advice can be challenged and the organization remains answerable for both.
Research references
- Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991–1006. View source
- Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. View source
- Wang, D.-Y., et al. (2023). Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation bias. Journal of the American Medical Informatics Association, 30(10), 1684–1692. View source
- Alon-Barkat, S., & Busuioc, M. (2023). Human–AI interactions in public sector decision making: Automation bias and selective adherence to algorithmic advice. Journal of Public Administration Research and Theory, 33(1), 153–169. View source
- Selten, F., Robeer, M., & Grimmelikhuijsen, S. (2023). Just like I thought: Street-level bureaucrats trust AI recommendations if they confirm their professional judgment. Public Administration Review, 83(2), 263–278. View source
- Berman, A., de Fine Licht, K., & Carlsson, V. (2024). Trustworthy AI in the public sector: An empirical analysis of a Swedish labor market decision-support system. Technology in Society, 76, 102471. View source
- European Union. (2024, consolidated 2026). Regulation (EU) 2024/1689, Article 14: Human oversight. View official text