A caseworker reads an application, forms a provisional view, and then sees an AI system recommend the opposite decision. She changes her answer. Was that automation bias? The change alone cannot answer the question. Perhaps the system noticed evidence she missed. Perhaps the recommendation was wrong but the file was ambiguous. Perhaps the interface made disagreement unusually costly. Measuring automation bias requires a task that distinguishes warranted correction from unwarranted deference, and a record of what the person could know at the moment of choice.
Automation bias has a recognizable public meaning: people may accept automated advice too readily. In research, the construct is narrower and harder to observe. The investigator must specify the system’s error, the human’s opportunity to detect it, and a comparison that shows how the aid changed behavior. A single agreement rate or trust rating is too blunt. This methods entry explains what different task designs measure, where their validity breaks, and how to read a result without turning one score into a diagnosis of a worker or institution.
Define the target before counting errors
Skitka, Mosier and Burdick studied decisions in simulated flight tasks with and without computerized monitoring and recommendations. Their work helped establish the distinction between commission errors, in which a person follows an incorrect automated recommendation, and omission errors, in which a person misses a problem that the aid failed to flag. Parasuraman and Manzey later reviewed this literature and integrated bias with attention and monitoring. Goddard and colleagues surveyed studies across fields, with particular concern for clinical decision support. These are different levels of evidence: controlled tasks, an explanatory review and a systematic review of heterogeneous studies.
The operational definition matters. If a clinician accepts a wrong diagnostic suggestion, the mistake may be consistent with automation bias, but it may also reflect limited information, time pressure or a competing diagnosis. A valid experiment needs an independently specified answer or adjudication procedure, a known error in the aid, and evidence that the participant had a feasible way to identify it. Without those conditions, a researcher’s label may merely rename disagreement with a later verdict.
Automation bias is not the same thing as algorithmic bias in the model’s training data. A model can perform unequally across groups even when every user checks it carefully. A user can over-rely on a uniformly inaccurate system. The two problems can interact, especially if errors concentrate where users expect the system to be reliable, but the measurement units differ. One asks about a system’s output and distribution of error; the other asks how human decisions change in response to that output.
Nor is automation bias identical to high trust. A participant can report distrust yet follow a recommendation because it is the only usable evidence in a rushed workflow. Another can express confidence in the system while independently checking a particular answer. The behavioral claim requires behavior measured against opportunities to accept, reject or seek other evidence. Survey responses may help explain that behavior but cannot replace it.
Build a task with visible opportunities to disagree
A basic experiment asks people to make decisions with or without an aid. Within aided trials, the system must sometimes be correct and sometimes wrong. If it is always correct, agreement is sensible reliance. If it is always wrong, participants may rapidly learn to ignore it, creating an artificial distrust task. Varying reliability can show whether participants adjust, but it also changes expectations over time. Researchers should describe the sequence, error frequency and what participants were told about system quality.
The most useful task makes the decisive evidence available to the human. In a simulated flight display, an instrument value might contradict a computer alert. In a document review, a date or rule might contradict a generated summary. The researcher must ask whether the evidence is actually legible under the imposed workload. A technically visible fact that cannot reasonably be found before the timer expires does not provide a fair test of vigilance.
Researchers also need an unaided baseline. Aided participants might make fewer errors overall even while committing some automation-induced errors. Conversely, an apparently cautious group may reject useful recommendations and perform worse. Comparing the same kind of item without the aid, or randomizing aid availability across comparable cases, helps separate susceptibility to a misleading cue from ordinary task difficulty. Counterbalancing item difficulty matters because a hard case assigned only to the aided group could make the aid look harmful for the wrong reason.
One useful sequence records an initial judgment, then shows the recommendation, then records a final judgment. An initial correct answer that becomes wrong after an erroneous recommendation is a strong behavioral indicator of harmful influence in that task. An initial wrong answer corrected by valid advice is beneficial reliance. A participant who remains wrong after the aid speaks may have ignored the aid or misunderstood both sources. These transitions are more informative than final accuracy alone, though asking for an initial decision can itself change how people use advice.
In some settings the aid is a monitor rather than an adviser. The human is supposed to notice events, and automation flags some of them. An omission error may appear when the system fails to flag a critical state and the person also misses it. That design must distinguish a general vigilance lapse from a lapse caused by reliance on the monitor. A no-aid comparison, a measure of inspection behavior and a clear account of workload make the interpretation stronger. Merely observing that a person missed an unflagged event cannot identify automation as the cause.
Separate commission, omission and useful reliance
A commission score often counts following an incorrect recommendation despite contrary information. The denominator should be the number of genuine opportunities to detect that error, not all trials. If a person saw ten wrong recommendations but only two included accessible disconfirming evidence, calling eight acceptances biased inflates the construct. If evidence was available only after the choice, the score measures exposure to a bad recommendation rather than failure of oversight.
Omission scores require similar care. A silent system may reduce a person’s monitoring because it usually detects anomalies, but it may also leave the person with too much to inspect. The researcher can test whether omission rises relative to a no-aid condition under comparable workload, and whether participants looked at the relevant evidence. Eye movement or click logs may illuminate attention, but looking at a value is not proof of understanding. Process traces are pieces of the explanation, not an unmediated view of thought.
Useful reliance belongs in the same table. A system might be right on most cases and reduce workload, allowing people to catch errors elsewhere. A study that counts only erroneous compliance portrays all influence as suspect. A balanced result reports correct acceptance, correct rejection, incorrect acceptance and incorrect rejection, with cases stratified by difficulty and system accuracy. It can then show the tradeoff between human effort and final decision quality instead of declaring every departure from an unaided answer a loss of agency.
Rates need uncertainty. If a study deliberately inserts only a few wrong recommendations, one additional error can sharply change a percentage. The analysis should report denominators and intervals, not merely a dramatic percentage. Clustered observations from the same person or scenario are not independent, and a polished dashboard can make repeated similar cases look like a large, diverse sample. Researchers should describe both participants and unique decision situations.
What common confounds look like
Task difficulty is the first confound. A participant may agree with a wrong AI answer because the case is genuinely hard, not because the output came from automation. An unaided comparison helps, but it must preserve information and timing. If the aided group receives a confident paragraph while the unaided group sees only raw files, the experiment changes more than source identity. If the objective answer rests on expert disagreement, a binary bias label may be particularly fragile.
Time pressure is another. A short deadline may make advice attractive because it saves search time. That can be a rational response to the imposed setting even when it occasionally produces errors. The study should distinguish a system that creates bad shortcuts from an organization that leaves no time for independent review. Reporting only an individual’s failure risks treating a design constraint as a personality trait.
Interface presentation can be a third. Position, color, default selections, confidence labels and required clicks all shape whether disagreement feels possible. A strong visual recommendation may dominate a faint source document. The experiment might therefore measure the combined effect of model and interface, not a general propensity to obey machines. That is still useful evidence if the deployed interface is the target. It is less useful for claiming a portable human tendency without specifying design conditions.
Expertise and accountability can complicate the picture. Experts may recognize some contradictions faster while still being vulnerable in unfamiliar subdomains. An instruction to justify disagreement may prompt better inspection or merely add paperwork. The reviews by Parasuraman and Manzey and Goddard and colleagues do not support a simple promise that training or warnings will erase the problem. A mitigation test must measure whether the actual decision improves and whether effort, delay or new errors appear elsewhere.
Finally, selection matters. Volunteers performing brief simulations may have different incentives, fatigue and responsibility from employees using a system over months. A laboratory manipulation can isolate a causal effect of a recommendation under controlled conditions. It does not automatically estimate the prevalence of harmful reliance in a public agency. Field observations capture real workflow but face stronger confounding. A good evidence program uses both and keeps their claims separate.
Read validity at several levels
Construct validity asks whether a score represents the intended phenomenon. A count of agreement with AI output fails this test unless the task distinguishes correct from incorrect advice and considers the human’s available evidence. A count of final errors is also insufficient unless it shows how automation changed those errors. A well-specified commission or omission measure does better, while still representing one task design rather than every form of over-reliance.
Internal validity asks whether the aid caused the observed difference. Random assignment, balanced cases and an unaided condition can help. The initial-final judgment design adds a within-person trace but may make participants unusually reflective. No design is free of tradeoffs. Researchers should state which inference is strongest: the effect of seeing a particular suggestion, the effect of a whole interface, or the difference between organizations with distinct workflows.
Ecological validity asks whether the task resembles consequential use. A caseworker may have colleagues, appeals, policy manuals and a backlog, none of which appears in a short online study. A professional may have to document a reason rather than click a label. When translating a result, readers should ask whether the study reproduced the information, discretion and consequences of the real task. Similar-looking screens do not guarantee similar decisions.
Measurement reliability asks whether the task yields interpretable patterns when cases, occasions and raters change. A score from three rare error trials is unlikely to support ranking individuals. Even if a group effect is clear, classifying one worker as “automation biased” may be unjustified. Replication across scenarios and transparent scoring rules improve confidence. The unit of inference should match the data.
A European oversight example
Article 14 of the EU AI Act explicitly refers to awareness of automatic or excessive reliance on outputs of high-risk AI systems, particularly recommendations for decisions by people. That legal text makes the research question operationally important. It does not supply a psychological test, nor does it make a particular training exercise sufficient for compliance. The legal requirement, the deployed system and the human task each need separate assessment.
A Swedish public body piloting a decision aid could predefine sample cases with independently adjudicated outcomes, including plausible model mistakes. It could compare unaided and aided decisions, record whether reasons were inspected, and ask staff where the workflow made dissent difficult. It should also assess whether model errors vary by applicant group. These are Institute design proposals derived from the measurement literature and regulatory context, not a tested Swedish protocol.
Such an evaluation should protect the staff and the people whose cases are used. A worker cannot demonstrate meaningful oversight if the organization punishes every disagreement with the model or denies time to verify facts. An appeal route for affected people matters even when measured error rates are low. Oversight is an institutional arrangement, not a score printed next to an employee’s name.
What the evidence does not show
The classic flight simulations do not show that all AI-supported public decisions are affected at the same rate. Reviews gather varied tasks and definitions; they do not turn a heterogeneous literature into one universal prevalence figure. A participant’s agreement with a wrong output is evidence of harmful influence only under the information and comparison conditions the study establishes. It is not a direct reading of their character, diligence or moral worth.
An error rate does not by itself distinguish automation bias from hard cases, poor system design, missing source material or impossible deadlines. A trust questionnaire does not repair that limitation. Neither a warning banner nor an instruction to “stay vigilant” is proven by this literature to solve every problem. Intervention claims need direct tests with appropriate controls and attention to the extra work they create.
The EU AI Act’s recognition of automation bias does not validate one instrument for every high-risk system. The Institute’s proposed evaluation table is a synthesis for local testing. It should be revised with workers, affected people, domain experts and the actual error profile of the system. If the model changes, the interface changes or the use context changes, the measurement question changes too.
Return to the caseworker
The caseworker’s changed answer becomes interpretable when we know the recommendation’s accuracy, what evidence she could inspect, how much time she had and what happened on comparable cases without the aid. If the AI corrected an initial mistake, the change may be welcome. If it pulled a correct judgment toward an error while contradictory evidence was accessible, the task has captured something closer to automation bias. Both outcomes can occur in the same workplace.
Good measurement makes this distinction visible. It counts mistakes without hiding useful help, and it studies the conditions that invite each response. The purpose is not to demand that a person defeat every machine error through willpower. It is to design decision support and oversight so that human judgment has the information, time and authority required to matter.
Research references
- Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991–1006. View source
- Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. View source
- Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. View source
- Goddard, K., Roudsari, A., & Wyatt, J. C. (2014). Automation bias: Empirical results assessing influencing factors. International Journal of Medical Informatics, 83(5), 368–375. View source
- European Parliament and Council. (2024, consolidated 2026). Regulation (EU) 2024/1689, Article 14: Human oversight. View source