A doctor says “seventy percent.”
A student says “I am almost certain.”
A witness chooses four on a five-point confidence scale.
The words sound like internal states. In research, they can become data only after a more difficult question is asked:
How does reported confidence relate to what actually happened?
Confidence ratings are one of the most common tools in metacognition research. Participants make a first-order judgment, such as choosing which stimulus was brighter, answering a memory question or solving a reasoning problem, and then report how confident they are that the judgment was correct.
Calibration concerns correspondence between those subjective reports and objective outcomes.
The appeal is obvious. Confidence gives researchers access to a second layer of cognition that accuracy alone cannot reveal. The danger is equally obvious. A number on a scale can look more precise than the psychological process that produced it.
What confidence ratings measure
A confidence rating is a report about a decision or state of knowledge.
Depending on the study, the rating may be numeric, verbal, binary or continuous. A participant may choose between “low” and “high,” select 1–4, use a 0–100 probability scale, or place a marker on a visual analogue line.
The same response format can serve different constructs.
A rating made immediately after a perceptual choice may index decision confidence. A rating made before an examination may be closer to a judgment of learning. A global rating such as “How good is your memory?” is different again.
Methodological interpretation begins by specifying what event the confidence report is about, when it is elicited and what criterion it will be compared with.
Without that, “confidence” is too broad to be a measure.
Calibration is not the same as discrimination
Two properties are often confused.
Calibration, in the strict sense, concerns whether stated confidence matches observed accuracy across repeated judgments. If a person's judgments made with 80 percent confidence are correct about 80 percent of the time, those judgments are well calibrated at that confidence level.
Discrimination or resolution concerns whether the person gives higher confidence to correct responses than to incorrect ones.
A person can be poorly calibrated in level but still discriminate well. For example, they may be systematically underconfident while nevertheless giving relatively higher confidence to answers that are correct.
Another person can have a mean confidence that matches mean accuracy while failing to discriminate trial by trial.
This is why one “metacognition score” cannot answer every question.
Overconfidence and underconfidence require an external criterion
Overconfidence is not loudness.
It is not assertive speech. It is not social dominance. In calibration research, it usually refers to confidence that is systematically higher than accuracy under the specified scoring rule.
Underconfidence is the converse.
Even these terms require care. A participant using a four-point ordinal scale has not literally supplied calibrated probabilities unless the scale is interpreted that way. The meaning of “high confidence” depends on instructions and response format.
Researchers can calculate mean confidence minus proportion correct when confidence is represented on a probability scale. They can create calibration curves or bins. They can use proper scoring rules such as the Brier score when probabilistic predictions are appropriate.
But every score bundles assumptions.
The method must fit the data rather than the data being forced into a familiar metric.
Local and global confidence are different evidence
A local confidence judgment refers to a specific trial or answer.
A global confidence judgment refers to a broader estimate of competence, such as predicted percentage correct across a test.
The two can diverge.
A learner may have reasonably good trial-level discrimination but an inaccurate global self-concept. A professional may know which decisions feel uncertain while still holding an inflated view of overall accuracy.
This distinction matters for interventions. Feedback that improves global calibration may not improve moment-to-moment monitoring, and vice versa.
It also matters for public interpretation. Saying that a group is “overconfident” without specifying whether the evidence comes from local judgments, global predictions or personality-like self-report invites conceptual drift.
Confidence is inferential, not a transparent gauge
Stephen Fleming's 2024 review argues that personal-level confidence can be understood as an inference informed by models of the world and of one's own cognitive system.
This helps explain why confidence can incorporate more than the sensory or mnemonic evidence directly supporting a choice. Prior beliefs, expectations, task structure, response history and beliefs about one's own ability can all contribute.
The implication for measurement is important.
A confidence scale does not simply open a window into an otherwise hidden probability variable. It records the output of a judgment process.
That process can be biased, strategically reported or influenced by instructions.
Confidence is therefore scientifically useful not because it is pure, but because its relation to performance can be modelled.
First-order performance can contaminate simple metacognitive measures
Imagine two participants.
One performs an easy task at 95 percent accuracy. Another performs a hard task at 65 percent. Comparing their raw confidence-accuracy correlations may tell us as much about the task difficulty as about metacognitive monitoring.
This is why metacognition research developed methods designed to adjust for first-order sensitivity and response bias, including meta-d-prime.
Confidence calibration measures still have value, especially when the research question is directly about probabilistic accuracy or overconfidence. But they should not automatically be interpreted as pure metacognitive ability.
Dobromir Rahnev's 2025 comparison of 17 metacognitive measures makes this problem explicit. The measures differ in properties and many remain influenced by task performance.
Measurement choice must therefore follow the inferential target.
Reliability is not guaranteed by many trials
A measure can show an effect at the group level and still be weak for ranking individuals.
Rahnev's 2025 analysis found generally high split-half reliability for many measures in the datasets examined but poor test-retest reliability for most. That pattern matters if a study wants to call a score a stable individual difference.
A confidence measure may be sensitive to task, session, stimulus set and current state.
This does not make it useless. It changes what can be claimed.
A state-sensitive measure can be excellent for detecting experimental manipulation while being inappropriate for selecting employees or diagnosing a person's enduring metacognitive ability.
Construct validity includes purpose.
The act of asking can change the thing being measured
Confidence ratings are often treated as passive observations.
Recent evidence complicates that assumption.
Double, Goldwater and Birney's 2025 preregistered category-learning study with 710 participants found that eliciting confidence ratings changed learning strategy and reduced rule-based learning relative to comparison conditions. Earlier work has also found cases in which confidence judgments alter speed or accuracy, although reactivity is not uniform across studies.
This creates a classic measurement problem: the instrument can intervene in the process it is meant to observe.
Researchers should therefore consider a no-rating control when reactivity is plausible, especially in learning tasks where repeated reflection may alter strategy.
The strongest conclusion is not “confidence ratings always distort cognition.” It is that measurement neutrality must be tested rather than assumed.
Calibration curves can reveal patterns averages hide
Suppose a participant is 60 percent confident on one set of items and 90 percent confident on another.
A single average confidence score erases whether higher confidence tracked higher accuracy.
Calibration plots can preserve more structure by comparing observed accuracy at different levels of stated confidence. They can reveal systematic overconfidence at the top of the scale, underconfidence on easier items, or compressed use of the scale.
Yet bins introduce their own decisions. Too few observations create noisy estimates. Arbitrary bin boundaries can change the appearance of calibration. Continuous approaches may be preferable when the dataset supports them.
The visual elegance of a calibration curve should not be mistaken for certainty about the underlying psychological process.
Scale design changes the behavior being recorded
A binary “confident/not confident” response is fast but coarse.
A 0–100 scale offers apparent granularity but asks participants to map subjective evidence onto numerical probability. Some participants use endpoints readily; others avoid them. Verbal labels may be interpreted differently across languages and cultures.
The scale can also interact with incentives. Participants may report strategically if confidence affects reward, reputation or feedback.
Cross-cultural work therefore requires more than translating labels. The response categories need evidence that they function comparably enough for the intended comparison.
This is one place where Northern European research and applied institutions should resist importing instruments unchanged simply because an English-language scale is widely used.
Calibration is useful when the consequence of uncertainty matters
Why measure confidence at all?
Because two people with equal accuracy can require different decisions if one knows where their errors are likely to occur.
A well-calibrated clinician may seek a second opinion when uncertainty is high. A student may allocate study time toward answers they recognise as fragile. A forecasting team may combine judgments more intelligently when probability estimates have known calibration properties.
But moving from laboratory calibration to applied decision quality requires additional evidence.
The cost of an error matters. Confidence thresholds interact with base rates. A person may be calibrated overall and still be dangerously miscalibrated on rare high-stakes cases.
Application therefore needs decision analysis, not only metacognitive measurement.
Confidence in AI-assisted work creates multiple calibration targets
Human-AI systems add layers.
A user can be confident in their own answer, confident in the AI's answer, or confident in their ability to tell when the AI is wrong. The AI may itself emit a probability, verbal hedge or no uncertainty signal at all.
These quantities are not interchangeable.
The Institute uses joint calibration as a synthesis label for the problem of aligning human confidence, system reliability and oversight behavior in a shared task. It is not presented here as a validated single measure.
A useful study might compare unaided confidence, assisted confidence and actual joint accuracy while tracking when the user accepts or rejects system advice.
The older confidence literature gives methodological tools for this work, but human-AI collaboration creates new dependencies that need direct validation.
Confidence can become performative in institutions
Many cultures reward certainty.
People who state probabilities carefully can sound weak next to colleagues who speak in absolutes. Leaders may learn that confidence attracts attention even when calibration is never checked.
A calibration framework changes the norm.
The question becomes not “Who sounded certain?” but “How did stated certainty correspond to outcomes over time?”
This is an Institute institutional synthesis, not a claim that calibration metrics should govern every workplace.
There are ethical concerns. Recording confidence can become surveillance. Scores can be misused to rank workers outside validated contexts. People may game reports when consequences attach to them.
Measurement should serve learning and decision quality, not create a new theatre of quantified self-certainty.
Base rates and asymmetric costs change applied interpretation
Calibration can look good overall while hiding the cases that matter most. If a rare condition has a high cost when missed, a decision system may need excellent sensitivity to uncertainty in that subset rather than attractive aggregate calibration. Likewise, a forecaster can be well calibrated across common events while systematically overconfident on extreme events.
Applied evaluation should therefore stratify where scientifically and ethically justified: by difficulty, class, consequence or decision context. This is not permission for fishing through subgroups until a desired pattern appears. It is a reminder that calibration is conditional on the event distribution being evaluated.
A change in prevalence can also change the practical meaning of confidence thresholds. Metrics that look stable in a controlled task may behave differently when base rates shift. For high-stakes use, calibration should be monitored in the environment where decisions are actually made.
What the evidence does not show
Confidence ratings do not directly measure truth.
Good calibration does not mean a person is highly accurate; someone can be well calibrated at low accuracy.
High accuracy does not guarantee good calibration.
A confidence-accuracy correlation is not a pure measure of metacognitive ability.
Test-retest stability cannot be assumed from internal reliability.
Confidence elicitation cannot be assumed to be non-reactive.
A scale validated in one task, language or population is not automatically valid in another.
And a person who speaks confidently in ordinary conversation has not thereby demonstrated measurable overconfidence.
Interpretation checklist
Before using confidence ratings, specify the first-order task, the timing of the rating, the response scale, the criterion for correctness and the intended construct.
Decide whether the primary question is calibration, bias, discrimination, metacognitive sensitivity or decision policy.
Check whether task performance differs enough across people or conditions to confound the measure.
Consider reactivity.
Report reliability appropriate to the intended use, not only internal consistency.
Avoid trait language unless stability has been demonstrated.
And when the stakes are applied, validate the measure in the applied population.
The number after the answer
A confidence rating can look like a small addition to a task.
One answer. One number.
But that number sits at the meeting point of evidence, self-model, scale interpretation and decision strategy.
Its scientific value comes from refusing to treat it as self-explanatory.
Confidence becomes useful when it is placed beside reality and the relationship between them is studied carefully.
Research references
Fleming SM. Metacognition and Confidence: A Review and Synthesis. Annual Review of Psychology. 2024;75:241–268. https://doi.org/10.1146/annurev-psych-022423-032425
Fleming SM, Lau HC. How to measure metacognition. Frontiers in Human Neuroscience. 2014;8:443. https://doi.org/10.3389/fnhum.2014.00443
Rahnev D. A comprehensive assessment of current methods for measuring metacognition. Nature Communications. 2025;16:701. https://doi.org/10.1038/s41467-025-56117-0
Double KS, Goldwater MB, Birney DP. Reactivity to confidence ratings: evidence of impaired rule-learning. Metacognition and Learning. 2025;20:8. https://doi.org/10.1007/s11409-025-09413-5
Double KS, Birney DP. Confidence judgments interfere with perceptual decision making. Scientific Reports. 2024;14:14133. https://doi.org/10.1038/s41598-024-64575-7