Calibrated Trust In Ai across high-risk AI systems, public-sector decision support, professional judgment and accountability

A police officer reads a risk recommendation before choosing where to deploy attention. A dermatologist sees an AI label beside a lesion image. A public employment caseworker receives a model-generated assessment of the support a jobseeker may need. In each setting, asking whether the professional “trusts AI” is too small a question. Trust might describe an attitude, an expectation of reliability, a decision to follow advice, or confidence that an institution can correct mistakes. Those are different things, and none alone tells us whether the person made a good decision.

Calibrated trust in AI means that confidence and reliance are proportionate to what a system can actually do in a defined task, under the conditions in which it is used. It requires people to use useful advice when it is warranted, reject misleading advice when it is not, and know when the available evidence is insufficient for either choice. In consequential settings, the calibration also has an institutional dimension: whether the system, workflow and organization merit the trust they solicit.

Trust is a relationship between a person, a system and a task

Lee and See’s influential review of automation described appropriate reliance as the design goal. Their account places trust in a context of limited knowledge: people often cannot fully inspect a complex automated system at the moment of decision. They must use available evidence about its performance, process and purpose. That framing is valuable because “more trust” is not automatically better. Increased trust in an unreliable system can worsen decisions; distrust of a capable aid can also lose useful information.

Calibration is therefore conditional. A professional might reasonably rely on a system for a narrow classification task while withholding reliance on its prediction outside the population on which it was evaluated. They might accept a recommendation when the input data are complete and challenge it when a crucial record is missing. They might use an AI-generated summary to locate source passages but refuse to treat the summary as a final judgment. One global trust rating cannot express all these distinctions.

The relevant unit of analysis is a decision opportunity. What did the person believe before seeing the advice? Was the advice correct in that case? Could the person inspect the evidence? Did they change their decision, and with what effect? A questionnaire about trust can add information about expectations, but it cannot substitute for these observations. Equally, a high rate of following AI advice might mean sound use of a strong aid, conformity under pressure, or simply that the advice agreed with what the person already thought.

Calibrated reliance is not the same as reported confidence

In a web-based study, 223 dermatologists classified skin-lesion images before and after seeing AI advice. Their accuracy improved by about one percentage point on average. The researchers measured subjective trust and several forms of behavioral reliance. Participants relied more on correct than incorrect advice, but the overall pattern included substantial self-reliance and comparatively low relative reliance on AI. This combination is more informative than a simple declaration that clinicians trusted or distrusted the system.

Even here, “appropriate” must be interpreted carefully. The study used image-classification tasks, not a full consultation involving history, examination, follow-up and patient preference. A small average gain does not establish that the same tool will improve clinical outcomes in routine care. The proportions of correct and incorrect AI advice in a study can also affect how often a participant has a chance to demonstrate appropriate reliance. A person who rarely disagrees with the model has fewer opportunities to reveal whether they could reject a specific error.

The findings show why calibration should be examined at both levels. At the individual level, compare changes when the model is right and when it is wrong. At the system level, check whether the aid is validated for the users and cases at hand, whether uncertainty is communicated, and whether disagreement is feasible under real workload. A questionnaire can measure confidence in the aid; behavior and outcomes show what that confidence did in the task.

Professional judgment can filter AI, but can also preserve a bias

Selten and colleagues tested a mock predictive-policing aid with Dutch police officers. The officers trusted and followed recommendations more when they confirmed their prior professional judgment. Explanations of the AI recommendation did not increase trust in this experimental setting. The result resists two easy stories. Officers were not simply following every machine suggestion. Yet professional discretion did not necessarily supply an impartial correction: if the initial judgment carried a bias, selectively accepting confirming advice could leave that bias intact.

Alon-Barkat and Busuioc examined public-sector decision support in three Dutch experimental studies. They did not find that an algorithmic prediction was followed more than equivalent human-expert advice in their initial comparison, contrary to a universal automation-bias story. Their work also studied selective adherence when advice aligned with stereotypes. A pattern in the first two studies did not replicate in a study with civil servants. That variation matters. A result from one population or task should not be promoted into a rule that all officials behave the same way.

These studies are useful precisely because they separate mechanisms. Following the machine because it is a machine differs from following advice because it confirms one’s expectation. The two can produce the same visible click. To tell them apart, experiments need comparisons with human advice, prior judgments, conflicting evidence and varied cases. In deployed public service, researchers also need to observe the workflow: what records are available, what deadlines apply, and whether challenging a recommendation is practically possible.

Institutional trustworthiness is not a feeling on a scale

A person can report confidence in a system that their organization cannot explain, audit or repair. Conversely, a skeptical employee may work within a carefully governed system and still choose to check every output. The Swedish Public Employment Service case analyzed by Berman and colleagues makes this distinction concrete. Their empirical study identified problems involving transparency, intelligibility and stakeholder involvement in the use of AI to assess jobseekers’ support needs. The case does not measure a universal trust deficit across Swedish public services. It shows why institutional design is part of whether reliance is warranted.

For a jobseeker, the relevant question is not whether a caseworker gives the model a favorable survey score. It is whether the assessment uses suitable evidence, can be explained in terms meaningful to the person affected, can be corrected when records are wrong, and leaves room for accountable human judgment. For the caseworker, the question includes whether training, time and authority support a real challenge to the output. Trust becomes misplaced if the interface invites a human signature but the institution provides no workable means to inspect or change the decision path.

Public-sector use also adds a legitimacy problem. People may need access to a service regardless of whether they would choose to trust the model. Their ability to contest an outcome must not depend on the confidence of the official who used it. A well-calibrated professional can still be embedded in a poorly governed institution. A well-governed institution can still have individual overreliance or underreliance. Both levels require evaluation.

High-risk oversight adds a legal floor, not a performance result

Article 14 of the EU AI Act sets out human-oversight requirements for high-risk AI systems, including the ability, as appropriate, to understand relevant capacities and limitations, monitor operation, interpret output and decide not to use, override or reverse it. The legal text is important context for European deployment. It is not an experiment showing that any particular interface, explanation or training program produces calibrated trust. Compliance claims and actual decision performance must be assessed separately.

In a high-risk workflow, an oversight plan should specify which errors matter, who can spot them, what evidence they can access and what action they can take. A visible “human in the loop” icon tells us very little about those abilities. The person may receive advice too late, face a workload that defeats careful review, lack the source data needed to challenge a result, or be unable to reverse a downstream action. Conversely, withholding useful AI information can also impose costs. Calibration asks how to preserve the benefits of aid while keeping an effective route to refusal and correction.

Law is one part of that route. Evaluation needs task-level studies and deployment monitoring, including disagreement cases, model errors, appeals and outcomes for affected groups. An organization cannot infer success from the fact that an authorized employee clicked “approve.” It should ask whether that employee could identify a wrong recommendation and act on the finding without unreasonable penalty or delay.

What a useful calibration study would count

First, record the human’s initial assessment before showing AI advice. Without it, a final agreement cannot show whether the AI influenced the decision. Second, construct or sample cases in which the model is right, wrong, uncertain and outside its validated use. Third, compare acceptance and rejection of advice in each category. Fourth, measure whether the final decision improves the outcome relevant to the task rather than only the rate of agreement with a model.

Fifth, test the interface and organization. Does access to source evidence help? Does an explanation clarify a real limitation or merely sound persuasive? How much time is available for review? Who bears responsibility when the recommendation conflicts with a professional’s observation? Can a case be escalated or an outcome appealed? These questions are especially important when the person affected has less power than the institution making the decision.

Sixth, follow use over time. People learn a model’s habits, and models or populations may change. A carefully calibrated response in a short study may degrade when routine use creates fatigue or when model performance drifts. Conversely, experience can teach users where an aid is genuinely strong. Periodic audits should therefore compare expected performance with observed performance in the live setting, not simply repeat a one-time trust questionnaire.

Finally, distinguish levels of error. A user might be well calibrated to the performance data supplied by a developer, yet those data may be unrepresentative. A system might perform adequately overall while failing for a subgroup. An institution might have a good model but a poor contestation process. Each failure calls for a different remedy. Asking individuals to “trust AI the right amount” cannot repair a broken evidence base or missing appeal route.

Design for disagreement, not just agreement

A practical evaluation should include cases in which the human and system disagree for different reasons. In one case the professional may know something the system could not observe. In another, the model may recognize a pattern the professional missed. In a third, both may be uncertain because the available information is weak. Treating every disagreement as a defect of trust destroys the possibility of learning which party had the better evidence. It also pressures people toward a superficially neat agreement rate that can hide mistakes.

For each disagreement, ask what information was available at the moment, what additional information could have resolved it, and whether the final decision remained revisable. The dermatologist’s disagreement with an image classifier cannot be interpreted without knowing whether the classifier’s answer was correct for that image. The public caseworker’s refusal of a recommendation may be appropriate if the record omits a recent change in the person’s circumstances. A police officer’s acceptance of confirming advice may look confident while merely repeating an initial assumption. The evidence needed to assess each choice differs.

The interface should make these differences visible without pretending that a longer explanation is always a better explanation. It can show source records, conditions under which model performance falls, and the path for escalation. It can let a professional record why advice was accepted or rejected. The organization can then review a sample of disagreement cases, including those in which the model was subsequently found to be right. This supports learning in both directions. It also makes it harder to confuse a high acceptance rate with good calibration or a low acceptance rate with safety.

That review should include the people affected by decisions wherever feasible. A jobseeker may know that a relevant event is missing from an administrative record, while the model and caseworker both lack it. A decision can be consistent with all information visible inside the organization and still be wrong for the person who receives it. Meaningful contestation is therefore an information channel as well as a fairness safeguard. It tests whether the institution can revise an apparently confident conclusion when better evidence arrives.

What the evidence does not show

The studies reviewed here do not establish one universal level of trust appropriate for all AI. Police vignettes, dermatology images and public employment assessments have different stakes, information and kinds of expertise. The dermatology study’s average accuracy gain is not proof of patient benefit. The Dutch police result does not imply that explanations never help; it reports a null effect for the explanations and task tested. The public-sector experiments do not prove that automation bias is absent everywhere, and the selective-adherence result was not uniform across all their studies.

The Swedish case study identifies institutional challenges in one deployment, not a numerical estimate of all public-sector AI trustworthiness. Article 14 describes oversight duties, not validated psychological measures. No single self-report trust score can establish whether a human used AI appropriately in a consequential decision, and no simple rate of following AI can establish whether an institution deserves trust.

Return to the decision

The police officer, dermatologist and caseworker all need more than a reassuring instruction to “use judgment.” They need accurate evidence about the aid’s limits, a workflow that allows genuine disagreement and a record of what happens when advice is accepted or refused. Calibrated trust is visible when reliance changes with the quality of the advice and the needs of the case. Trustworthy governance is visible when people affected by a decision can understand and challenge its path. Both are continuing practices, not a label conferred by the presence of a human reviewer.