A participant looks at a headline for six seconds.
They have never seen the publication before. The topic is plausible but unfamiliar. There is a photograph, a source label and a sentence written to resemble ordinary news.
They choose "false."
The item is actually true.
What, exactly, has just been measured?
Perhaps the participant was poor at distinguishing reliable from unreliable information. Perhaps they were generally suspicious. Perhaps they distrusted the source. Perhaps the headline depended on knowledge they did not have. Perhaps the visual design looked synthetic. Perhaps the experiment trained them to expect deception simply by announcing that misinformation would be present.
This is the central difficulty in measuring misinformation susceptibility.
The response is observable. The psychological process behind it is not automatically identified by the response.
Definition
For this Knowledge Library, misinformation susceptibility measures are instruments or experimental paradigms designed to estimate how accurately people evaluate, discriminate, believe, reject or intend to share information that varies in veracity.
The plural matters.
Researchers may measure:
- belief accuracy;
- ability to distinguish true from false headlines;
- detection of false material;
- acceptance of true material;
- sharing discernment;
- confidence in judgments;
- response bias toward believing or distrusting information;
- memory for corrections;
- recognition of manipulation techniques.
These outcomes are related, but they are not interchangeable.
A person can be reluctant to believe false news by being reluctant to believe almost anything. That is not the same as being good at discernment.
The measurement target is usually discrimination
Many misinformation studies present participants with a mixture of true and false claims or headlines and ask for an accuracy, truth, belief or sharing judgment.
A basic discernment score can be computed as the difference between ratings of true and false items.
This is useful because it asks whether the participant treats reliable and unreliable information differently.
But a difference score can hide important structure.
Two participants can have the same discernment score for very different reasons.
One may correctly accept most true items and reject most false ones.
Another may reject nearly everything, including a large share of true items.
If an intervention makes people more skeptical across the board, false-item acceptance may fall while true-item acceptance also falls. A headline-level average may make that look like improvement unless sensitivity and bias are separated.
Sensitivity and response bias should be distinguished
Signal detection theory provides one useful framework.
Sensitivity asks how well a person distinguishes signal from noise, or in this context reliable from unreliable material.
Response bias asks where they place the decision threshold. Are they generally inclined to say "true"? Generally inclined to say "false"? Does the intervention move the threshold without improving discrimination?
This distinction matters because broad skepticism is not the same as accuracy.
A society in which nobody believes anything has not solved misinformation.
It has created a different epistemic problem.
The Misinformation Susceptibility Test is an important attempt at psychometric precision
The Misinformation Susceptibility Test, or MIST, was developed by Rakoen Maertens and colleagues to provide a psychometrically validated measure of news veracity discernment.
The validation program used seven independent samples with a combined N of 8,504 and produced 20-item, 16-item and 8-item forms. The authors explicitly separated veracity discernment from two judgment tendencies they describe as distrust and naivete.
That separation is one of the instrument's major conceptual strengths.
It recognizes that a person can miss misinformation through credulity and can also miss reliable information through excessive distrust.
The MIST also used item-generation and psychometric methods to create scalable forms that can be administered quickly.
But psychometric validation is not the same as universal validity.
A validated instrument is valid for claims supported by its validation evidence
Validity is not a permanent property stamped onto a test.
It is an argument about whether particular interpretations of scores are justified for particular uses.
A MIST score can support claims about performance on the kinds of news-veracity items represented in the instrument and populations studied during validation and later use.
It does not automatically support claims about:
- every form of misinformation;
- deepfakes or synthetic audio;
- private messaging environments;
- health rumors;
- wartime propaganda;
- highly technical scientific claims;
- all languages and cultures;
- susceptibility months or years later;
- behavior under social pressure;
- what a person actually shares in a live platform environment.
Those require additional evidence.
Content knowledge is never entirely absent
A truth judgment is partly a reasoning task and partly a knowledge task.
If a headline concerns a subject a participant knows well, they may evaluate it using stored knowledge.
If it concerns an unfamiliar subject, they may lean more heavily on source cues, plausibility, style or general trust.
That means performance can vary with topic familiarity.
A measure intended to assess general misinformation susceptibility should therefore avoid becoming a disguised current-affairs quiz.
It also needs item pools that do not become stale as factual circumstances change.
A headline that was false in 2023 may describe something that later becomes true. A source that was obscure may become familiar. An AI-generated image style that once looked suspicious may become ordinary.
Measurement in a moving information environment requires maintenance.
Real and false items need comparable difficulty
If false headlines are more sensational, less grammatical or less plausible than true headlines, the measure may partly test surface oddness.
If true items come from familiar institutions while false items come from invented domains, source familiarity may dominate.
If one set concerns politics and the other science, topic effects become entangled with veracity.
Good item construction therefore tries to match true and false materials on characteristics that should not determine the answer.
This is difficult.
It is also why researchers should report how items were selected, pretested and updated.
AI-generated items solve one problem and create another
Generative models can produce large item pools with controlled style and topic.
That can reduce dependence on a small set of viral headlines that participants may already have encountered.
The MIST development work used a neural language model as part of item generation.
But synthetic item generation creates its own validity questions.
Do generated headlines resemble misinformation people actually encounter?
Are they too cleanly balanced?
Does the model produce subtle stylistic regularities that participants learn?
Are human raters able to establish the ground truth confidently?
AI can make item production easier without making ground-truth construction trivial.
Belief and sharing are different outcomes
A person can believe a claim and choose not to share it.
They can also share something they doubt because it is funny, identity-affirming, socially useful or emotionally provocative.
Research by Pennycook and colleagues has repeatedly shown that sharing intentions can diverge from accuracy judgments.
This is why a measure of belief susceptibility should not be presented as a direct measure of sharing behavior.
Likewise, an intervention that improves sharing discernment may work by changing attention at the moment of sharing rather than changing underlying belief.
The dependent variable has to match the claim.
Confidence adds a second layer
Two people can make the same number of correct judgments but differ sharply in calibration.
One may know when they are uncertain.
The other may be equally accurate but highly confident in mistakes.
For real-world decision-making, that difference matters.
A useful misinformation assessment can therefore pair accuracy judgments with confidence ratings and examine whether confidence tracks correctness.
But confidence measures are not neutral windows into the mind. Asking for confidence can change how carefully people think, and confidence itself can depend on response scales, familiarity and domain.
The measurement layer should be interpreted as another behavior, not as direct access to certainty.
Cross-cultural generalization is a major open problem
Information ecosystems differ by language, journalism norms, platform use, institutional trust and political history.
An instrument built from English-language headlines in the United States or United Kingdom may not transport cleanly into Sweden, Finland, Poland or India.
Translation alone is not enough.
The item must remain comparable in difficulty and meaning after translation. The source cues must make sense locally. The factual content must not privilege one population's knowledge.
European and Nordic use therefore requires local validation rather than decorative localization.
The relevant question is not, "Can this test be translated into Swedish?"
It is, "Does the translated instrument support the same score interpretation in the population where it will be used?"
Susceptibility is partly situational
A laboratory task often gives the participant one job: judge accuracy.
Real platforms give them many jobs at once.
They are talking to friends, managing identity, scanning quickly, reacting emotionally, following links, switching tasks and responding to recommendation systems.
Accuracy may not be the active goal.
This is one reason accuracy prompts can improve sharing quality under some conditions: they change what is salient at the moment of choice.
The implication for measurement is important.
A person who performs well when explicitly instructed to judge truth may still behave poorly when truth is one goal among many.
Dispositional capacity and enacted behavior should not be collapsed.
Longitudinal stability also needs care
If someone scores poorly today and better next month, several explanations are possible.
They may have learned.
The item set may be easier.
They may remember specific answers.
The news environment may have changed.
They may be more motivated on the second occasion.
Test-retest reliability is therefore necessary but not sufficient. Repeated measurement needs alternate forms and clear decisions about whether the construct is expected to be stable or trainable.
A measure of susceptibility cannot simultaneously be treated as a fixed trait and as an intervention-sensitive skill without explaining the timescale.
Population profiles are not destiny
Large datasets can identify statistical associations between misinformation performance and age, education, cognitive reflection, media habits or other variables.
Those relationships can help researchers understand heterogeneity.
They should not be turned into profiles of who is "gullible."
Group averages do not determine individuals, and susceptibility depends on topic, context and exposure.
A measurement system becomes ethically poor when it converts a bounded test result into a moral identity.
Item-level accuracy can conceal person-level strategy
A test score usually compresses many decisions into one number.
That compression can be useful, but it can also erase the strategy that produced the score.
Suppose two participants each answer 15 of 20 items correctly.
One confidently distinguishes almost every clear item but struggles with ambiguous headlines. The other uses a blanket rule that anything from an unfamiliar outlet is probably false. Their total score can match even though their epistemic behavior is very different.
For research that aims to understand mechanisms, item-level models are therefore often more informative than raw totals. Researchers can examine item difficulty, person ability, confidence, response tendency and interactions with topic familiarity. Psychometric tools such as item-response modeling can help determine whether the test behaves similarly across groups and whether particular items function differently for reasons unrelated to the intended construct.
This is especially important when a measure is used across countries, languages or political contexts. An item that is easy in one country may be obscure in another. A recognizable institution in one language community may be unknown elsewhere. Without testing measurement invariance or differential item functioning, group-score differences can be misread as differences in susceptibility when they partly reflect the instrument.
Ground truth is an epistemic decision too
Every misinformation test depends on a procedure for deciding which items count as true and which count as false.
That procedure is often less visible than the participant scoring, but it is foundational.
Some claims can be verified against stable records. Others involve early scientific findings, estimates, causal interpretations or events that are still unfolding. A headline can also contain a mixture of accurate facts and misleading implication.
Researchers therefore need transparent ground-truth procedures: source hierarchies, expert adjudication where necessary, dates of verification, rules for disputed claims and a process for updating items when facts change.
This becomes more difficult with health, geopolitical and scientific material, where the best available answer may legitimately change as evidence accumulates.
Calling an item “false” does not make the classification self-validating. The label itself needs provenance.
Reliability and validity answer different questions
A measure can be reliable without measuring what researchers think it measures.
If participants consistently distrust every headline, their responses may be internally consistent. That does not mean the score validly captures discernment.
Reliability concerns consistency. Validity concerns the interpretation and use of scores.
For misinformation measures, useful validity questions include whether scores predict performance on new items, whether they relate to theoretically relevant constructs without collapsing into them, whether results survive changes in format, and whether the measure predicts meaningful behavior outside the test.
Convergent evidence can come from associations with related capacities such as cognitive reflection or source evaluation. Discriminant evidence asks whether the measure remains distinct from general knowledge, political trust or reading ability. Predictive evidence asks whether a score tells us anything about future or real-world information behavior.
No single coefficient answers all of these questions.
Repeated testing can change the thing being measured
Misinformation paradigms are sometimes treated as passive observation.
Repeated exposure to true and false items, feedback, confidence ratings and explicit instructions can themselves teach participants what kind of task they are in.
This creates several possibilities.
Participants may become more skeptical because they infer that false items are common. They may learn stylistic cues that distinguish generated false items from true journalism. They may remember specific headlines when retested. Confidence elicitation may alter attention and later memory.
Longitudinal studies therefore need to distinguish genuine change in discernment from familiarity with the instrument.
Alternate forms, delayed testing and pre-registered scoring rules can help. So can measuring whether improvement transfers to novel topics and formats rather than only to repeated items.
Intervention evaluation should report both gains and error trade-offs
When a media-literacy or prebunking intervention is evaluated with a misinformation measure, “improvement” should be decomposed.
Did participants become better at rejecting false material?
Did they also become worse at accepting true material?
Did confidence become better calibrated?
Did the effect persist after the intervention?
Did it transfer to unfamiliar items?
Did it change sharing decisions as well as belief ratings?
These questions matter because an intervention that makes everyone more suspicious can look successful if only false-item rejection is reported.
A high-quality evaluation should therefore report true-item and false-item performance separately where possible, along with a discernment metric and an interpretable measure of response tendency. It should also disclose item-selection procedures and whether analysts were blind to condition when classifying ambiguous responses.
The more public importance attached to a “misinformation susceptibility score,” the more important these details become.
What the evidence does not show
Existing measures do not show that misinformation susceptibility is one unitary trait.
They do not establish that performance on headline-veracity tasks predicts behavior across all real-world platforms and topics.
They do not show that low belief in false items necessarily reflects good discernment rather than generalized distrust.
They do not establish equal validity across languages, countries, age groups or information environments without local evidence.
They do not show that one short score can stand in for source evaluation, confidence calibration, verification behavior, media literacy and social motivation all at once.
And no instrument eliminates the need to inspect the items and decision rules through which its score was produced.
What this changes for Inner Technology
For the Institute, misinformation measurement is not a search for a single number called discernment.
It is an exercise in measurement humility.
If the capacity is discrimination, the measure should separate discrimination from blanket skepticism.
If the capacity is verification, the task should observe verification behavior rather than infer it from a final answer.
If the capacity is calibration, confidence should be compared with correctness.
If the claim concerns real-world use, the design needs ecological evidence.
This makes measurement itself part of epistemic agency.
The question is not merely whether a person can answer a test.
It is whether the test can justify what institutions want to conclude about the person.
The headline again
The participant chose "false."
The item was true.
That single error tells us almost nothing by itself.
A serious measure asks what pattern the error belongs to, what alternatives were available, how the items were built, what the participant knew, how confident they were and whether the score travels beyond the test.
The point is not to make measurement impossible.
It is to make the inference proportionate to the evidence.
Research references
- Maertens R, Gotz FM, Golino HF, Roozenbeek J, Schneider CR, Kyrychenko Y, Kerr JR, Stieger S, McClanahan WP, Drabot K, He J, van der Linden S. The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment. Behavior Research Methods. 2024;56:1863-1899. https://doi.org/10.3758/s13428-023-02124-2
- Pennycook G, Epstein Z, Mosleh M, Arechar AA, Eckles D, Rand DG. Shifting attention to accuracy can reduce misinformation online. Nature. 2021;592:590-595. https://doi.org/10.1038/s41586-021-03344-2
- Pennycook G, Rand DG. Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation. Nature Communications. 2022;13:2333. https://doi.org/10.1038/s41467-022-30073-5
- Saltor J, et al. Thinking disposition, thinking style, and susceptibility to causal illusion predict fake news discriminability. Applied Cognitive Psychology. 2023;37(2):360-368. https://doi.org/10.1002/acp.4008
- Kyrychenko Y, Koo HJ, Maertens R, Roozenbeek J, van der Linden S, Götz FM. Profiling misinformation susceptibility. Personality and Individual Differences. 2025;241:113177. https://doi.org/10.1016/j.paid.2025.113177
- Fischer H, Huff M. Metacognition as a target of science communication. Nature Reviews Psychology. 2025;4:554-555. https://doi.org/10.1038/s44159-025-00478-2