A clinician uses an AI tool that is accurate for one class of images but unreliable for another. Trusting it equally in both cases would be a mistake; ignoring it entirely might also waste useful information. Trust calibration concerns the match between a person’s trust in a system and the system’s demonstrated capability in the relevant context. The practical aim is appropriate reliance: use help when it improves a decision and retain scrutiny where it can fail. This is a relationship between evidence, task, and behavior, not a campaign to make users trust technology more.
Trust, reliance, and calibration
Trust is an attitude or expectation about another agent or system. Reliance is an observable choice to use its output. A person may report high trust but check every recommendation because the stakes are high; another may report skepticism yet follow a recommendation under time pressure. Lee and See’s influential review connects trust with the design of appropriate reliance on automation. It emphasizes that use should be sensitive to performance, process, purpose, and context. A trust questionnaire alone therefore cannot establish that reliance was appropriate.
Calibration can be thought of as correspondence between confidence and actual performance, but the idea needs a reference level. A tool might be 90 percent accurate overall while failing disproportionately on rare but important cases. A user who trusts it “90 percent” is not necessarily calibrated if they cannot identify those failure modes. The relevant capability may depend on image quality, patient group, or task stage. A meaningful study compares behavior with the tool’s performance on the cases where the user made a decision.
Overtrust can produce misuse: following an output when the system is wrong or outside its intended scope. Undertrust can produce disuse: ignoring a useful output. Neither is captured by how frequently the tool is opened. A user may rely heavily on an excellent system and be well calibrated; they may rely rarely on a poor system and also be well calibrated. The objective is not a middle amount of use but decisions that reflect the system’s actual strengths and limitations.
Trust calibration also differs from trustworthiness. A system provider can supply evidence about validation, monitoring, and limitations; a user forms expectations from that evidence and experience. If the system is poorly tested, trying to “calibrate” the user through reassuring language is ethically backward. Improving the system, documenting uncertainty, and establishing oversight may be necessary before any interface intervention can support appropriate reliance.
How trust develops and changes
Trust can be local to a function rather than global to a brand. A user might trust a navigation system to estimate travel time but not to identify road hazards, or trust a language model to summarize a document but not to supply a legal citation without checking. Asking for one overall rating hides these distinctions. Studies can describe separate capabilities and ask users to predict performance for each. Calibration improves as the question becomes tied to the decision the person actually faces.
Hoff and Bashir reviewed empirical factors that influence trust in automation, including person-level tendencies, situational demands, and learned experience with a system. Initial impressions can be shaped by appearance, authority, and organizational endorsement. Later errors or successes may shift trust, but users do not always update perfectly. A system that performs well in routine cases may earn trust that carries into cases where it is less capable. Longitudinal evaluation is therefore important.
The pattern of failures matters, not only their count. A predictable limitation that is well communicated may be easier for users to manage than an occasional confident error with no warning. False alarms can encourage people to ignore alerts; missed cases can create a different kind of risk. Studies should distinguish these error types and measure whether users notice them. A broad average accuracy figure can hide the particular event that determines whether reliance was warranted.
Dzindolet and colleagues studied how trust relates to automation reliance. Their work illustrates that people may use performance feedback and explanations when deciding whether to rely on an aid, but the design of the task shapes the result. A brief experiment cannot establish how trust evolves in a busy workplace over months. It can identify mechanisms to test in deployment. A trustworthy interface should help users learn limitations rather than only display confidence-building cues.
Institutional context also shapes trust. If staff are evaluated on speed, they may accept AI suggestions even while doubting them. If responsibility for errors is unclear, they may avoid a tool that would help. Observed reliance can therefore reflect incentives and workflow as much as belief. An evaluation that only surveys user attitudes misses these pressures. Calibration requires analyzing the full decision environment, including time, accountability, training, and available alternatives.
Measuring appropriate reliance
One common measure counts how often participants switch from their initial answer after seeing AI advice. Switching is not inherently good or bad. A switch from a correct human answer to an incorrect AI answer is overreliance; a switch from an incorrect human answer to a correct AI answer may be beneficial. Staying with an initial answer also has two possible meanings. Analyses should separate these cases and report joint accuracy, not only the overall rate of accepting advice.
Case selection matters. If most test cases are easy for both human and AI, an interface may appear safe while revealing little about complementarity. The most informative cases often involve disagreement or a known model failure. However, a test set composed only of unusual failures may exaggerate everyday risk. Researchers can report performance separately by case type and then weight outcomes to match intended use. This makes the tradeoff between missed benefit and harmful reliance visible.
A useful experiment includes cases where the AI is correct and cases where it is wrong, ideally with known ground truth. Researchers can observe whether people accept correct advice and reject incorrect advice. They can compare human-alone, AI-alone, and human-AI performance. But these comparisons can be distorted if case difficulty differs or if participants can identify errors for reasons unrelated to the interface. The test set should reflect intended use and include meaningful failure modes.
Calibration can be assessed across levels of confidence and task type. A user might respond appropriately when the model signals uncertainty but overrely on high-confidence errors. Researchers can examine whether the user’s decisions track actual conditional accuracy. They should avoid equating a confidence number with a guarantee: machine scores may themselves be uncalibrated or shift after deployment. If the AI’s confidence is unreliable, displaying it can mislead rather than help.
Feedback can improve learning about a tool, but its form matters. Telling users that the system was wrong after each trial may change behavior in a laboratory; in real practice, ground truth can arrive late or remain disputed. A user may never know which recommendations were mistaken. Monitoring and audit processes are thus part of calibration. Without them, users learn from a biased sample of visible outcomes and may misjudge performance.
Explanations do not automatically solve overreliance. Bansal and colleagues examined how AI explanations affect complementary team performance and found that explanations can have complicated effects rather than simply making a human-AI team better. An explanation may sound plausible for a wrong recommendation. The relevant test is whether it helps users distinguish correct from incorrect outputs and improves joint performance, not whether participants say they liked it.
Interface and workflow design
Confidence displays can be useful only when the number has a clear meaning and stable calibration. A score of 0.9 might be a class probability, a ranking score, or a vendor’s opaque certainty label. The user needs to know which. If a model was calibrated on a population unlike the current one, the number may be unreliable. A design team should validate the score on the deployment distribution and show limitations in language that supports decisions rather than dressing the number as a universal assurance.
A well-designed interface can show what evidence the system used, what it did not see, and where validation is weak. It can make uncertainty legible without burying the user in technical detail. Yet an uncertainty warning that appears on every case may be ignored, and one that appears too rarely may create false confidence. Warnings should be tested for comprehension and decision impact. The wording should support action, such as checking a source image, rather than merely transferring liability to the user.
Design can also make independent judgment possible. If a reviewer sees the AI answer before examining the evidence, the suggestion may anchor their interpretation. In some tasks it may be useful to ask for a preliminary human assessment first, then show the AI output for comparison. This can reduce uncritical acceptance but may cost time. The tradeoff should be tested in the intended workflow, including whether delayed presentation causes users to ignore useful advice.
Human expertise matters. A novice may benefit from a different level of explanation and training than an experienced specialist. Experts can identify some AI errors more readily, but can also be influenced by a persuasive interface. A one-size-fits-all trust score is unlikely to capture these differences. Evaluations should sample the people who will actually use the system and measure whether the design supports them on their actual tasks.
Escalation routes are part of appropriate reliance. A user who sees an uncertain or contradictory output needs a way to inspect primary evidence, consult another person, or defer a decision. If the workflow offers only “accept” and “reject” under a tight deadline, calibration is constrained regardless of the user’s attitude. Organizational design can either enable scrutiny or make overreliance the path of least resistance.
Consequences and accountability
Auditing after deployment is especially important because tasks and data change. A document classifier trained on one policy version may perform worse after a rule change; an image model may face new equipment or patient populations. Users cannot calibrate well to a hidden shift they never observe. Organizations should log performance, review contested cases, and update training and interfaces when failures emerge. Calibration is maintained through governance, not achieved once during onboarding.
High-stakes decisions require attention to unequal errors. A model that performs well on average may be less reliable for a subgroup or unusual case. Calibrated trust should be assessed at that level when evidence permits. Users need accurate scope information, and organizations should monitor performance after deployment. A prompt that tells a clinician to “use judgment” is not enough if relevant limitations are hidden or if the system’s quality drifts.
Responsibility cannot be assigned to the human alone simply because a human clicked the final button. Developers, deployers, managers, and regulators shape what information the user sees and how much time they have. A calibrated human-AI relationship depends on those upstream choices. The person at the interface should have authority to challenge a recommendation and access to a meaningful alternative. Otherwise apparent trust may be compliance with a constrained process.
The concept also applies outside health care. A teacher evaluating AI-generated feedback, a public servant reviewing eligibility suggestions, or a traveler following navigation advice must judge whether the tool fits the case. Each setting has different stakes and ways to verify outcomes. Evidence from one domain should not be transferred automatically to another. The common question is whether reliance tracks demonstrated capability in the actual context.
What the evidence does not show
It also does not show that trust should be high whenever a system is technically advanced. A simpler tool with transparent, stable limits may support better decisions than a more complex one whose errors are hard to detect. Evaluation should begin with the decision problem and credible alternatives, then ask how the tool changes outcomes for the people who must use it.
The evidence does not show that increasing self-reported trust improves decisions, that a confidence display is always useful, or that explanations automatically produce calibrated reliance. A system’s average accuracy is not enough to judge a difficult case or subgroup. Laboratory improvements may not persist in deployment. Appropriate reliance requires ground truth where possible, failure-mode testing, monitoring, and organizational conditions that allow the user to act on doubt.
The clinician’s task is not to decide whether AI is generally good or bad. It is to determine when this tool’s output deserves weight and when independent evidence must lead. Trust calibration offers a way to ask that question precisely, with attention to the system’s performance and the human’s real opportunity to verify it.
Research references
- Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46, 50–80. View source
- Hoff, K. A., & Bashir, M. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57, 407–434. View source
- Dzindolet, M. T., et al. (2003). The role of trust in automation reliance. International Journal of Human-Computer Studies, 58, 697–718. View source
- Bansal, G., et al. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. Proceedings of CHI 2021. View source
- Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5, 1–21. View source