A reader is shown twenty headlines and marks six false ones as accurate. Is the reader unusually susceptible to misinformation? The answer depends on what else was shown, how the question was asked, and what “susceptible” is meant to predict. If the same reader also rejects nearly every true headline, the six endorsements may reflect a broad tendency to distrust or a particular response scale rather than an inability to distinguish truth from falsehood. If the false headlines had appeared repeatedly before the test, the score may partly reflect familiarity. Measurement begins by refusing to collapse those possibilities into one label.
In research, misinformation susceptibility is often operationalised through judgments about the accuracy or credibility of claims whose truth status has been independently established. This is useful, but it is a family of tasks rather than a directly observed inner trait. The unit being measured may be a response to a particular headline on a particular day. A claim that the task measures a stable person’s susceptibility requires additional evidence about reliability, generalisation, and prediction. The distinction matters when results are used to evaluate an intervention, compare groups, or make claims about how people navigate a changing information environment.
What is being measured?
The narrowest outcome is false-item endorsement: the share of false claims a participant rates as true, credible, or likely accurate. That outcome is important because endorsing a false claim is one form of error. It is not, by itself, a measure of truth discernment. Discernment concerns the separation between responses to true and false items. Someone who calls everything true can endorse many false items while also identifying many true ones. Someone who calls everything false can avoid false endorsements while failing to recognise true information. Neither pattern is described adequately by a false-item score alone.
A study can report the proportion of true items accepted, the proportion of false items accepted, and their difference. When individual binary judgments and sufficient trials are available, signal detection methods can separate sensitivity to veracity from a response criterion, although their assumptions and estimation choices still require scrutiny. Sensitivity asks whether true and false items receive different responses. Criterion asks whether a participant is generally liberal or conservative in saying “true.” A 2022 experiment on time pressure used this framework and found reduced discrimination without a corresponding shift in response bias in its tested headline task. That is a result about that experimental context, not a universal effect of time pressure.
Confidence, willingness to share, recall, correction acceptance, and behaviour are different outcomes again. A person may suspect that a headline is inaccurate yet intend to share it for humour, identity expression, or discussion. A person can give a confident rating while later changing it after checking a source. A test of immediate headline judgment does not directly observe the provenance checks, conversations, feeds, and institutional cues that shape real-world action. A good instrument names its outcome precisely rather than treating every response as interchangeable evidence of belief.
Construct validity starts with the proposed use
Construct validity is the case for interpreting scores in a particular way and using them for a particular purpose. It is not a permanent property conferred by a catchy test name. Researchers need to say whether they seek to rank people within one sample, detect change after a short intervention, compare languages, or estimate performance in a specific information setting. Each use asks for different evidence. A short task that discriminates participants in one study may be too noisy for individual assessment, and a task sensitive to one topic may not generalise to other topics.
The Misinformation Susceptibility Test, or MIST, is a useful example of deliberate instrument development. Its authors generated and selected news items and evaluated short forms with psychometric methods across multiple samples. That work provides stronger evidence than simply assembling a handful of striking headlines. It does not make the items culturally timeless or prove that a score measures every form of misinformation response. The published paper frames the test as a measure of news veracity discernment; that construct is narrower than all possible meanings of “susceptibility.” Its development history also shows why item selection, difficulty, and reliability need to be reported rather than assumed.
Before using any test, define the target universe of claims. Does it concern political news, health advice, commercial offers, manipulated images, rumours in a workplace, or general factual statements? Is the intended task rapid recognition, considered evaluation, or a decision after access to external sources? These are not interchangeable settings. The same person can perform differently because topic knowledge, source cues, stakes, and opportunity to verify differ. An instrument can be valid for one question and weak for another.
Item sampling is part of the result
Headlines are not neutral containers for a truth label. They differ in plausibility, emotional tone, length, novelty, topic, source, image, and the extent to which their truth depends on a date or place. A set of obviously absurd false items paired with familiar true items may make a task easy for reasons that do not generalise to subtle falsehoods. Conversely, obscure true items can make participants look indiscriminately sceptical. Reporting only a participant average conceals this item structure. Researchers should describe how items were sourced, how truth was established, when that status was checked, and how difficulty was balanced or modelled.
Truth status also needs care. Some public claims are partly true, conditional, or later superseded. A headline can accurately report what a person said while the quoted claim is false. A changed scientific estimate can render yesterday’s apparently clear answer incomplete. An item set should avoid turning genuine uncertainty into a forced binary where possible, or else state that the score concerns agreement with the selected answer key under a defined evidentiary standard. The answer key is a research decision, not an invisible property of the task.
The balance of true and false items changes the meaning of raw accuracy. In a test with mostly false items, a participant who says “false” to everything can appear accurate without discerning much. In a balanced test, overall percentage correct is easier to interpret, yet it still merges two kinds of error. It is therefore helpful to publish results for true and false items separately, alongside a discernment measure and a response-bias estimate where appropriate. This makes it possible to see whether an apparent improvement came from better discrimination or merely from a more sceptical threshold.
Item-level analysis matters for group comparisons. If one group has more prior exposure to a claim, more relevant knowledge, or a different interpretation of wording, a score difference need not represent a general difference in susceptibility. Translation can change idiom or perceived source credibility. Claims about demographic or national differences require evidence that the items function comparably across those groups. A systematic meta-analysis of news judgments illustrates how overall discernment and tendencies to doubt true news can vary with item sets and study context. It is a reminder to examine the sampling frame before giving a score a broad social meaning.
Response format can create apparent susceptibility
“Is this accurate?”, “Have you heard this?”, “Would you share it?”, and “How credible is it?” invite different mental operations. Hearing a statement before is not equivalent to believing it. Sharing intention can reflect incentives other than conviction. A credibility scale may capture uncertainty differently from a forced true/false choice. Even the presence of a “not sure” option changes how cautious respondents can express themselves. If a study changes the format between groups or before and after an intervention, the change may alter the construct being measured.
Thresholds matter as well. Suppose a respondent moves from rating false claims “probably true” to “unsure” while rating true claims similarly. Counting only definite true responses could make false-item endorsement fall, but the overall ability to separate classes might remain unchanged. Researchers should preserve enough response detail to inspect distributions and should explain how ordinal responses were dichotomised, if they were. A single threshold chosen after looking at results can magnify an apparent gain.
Researchers sometimes use reaction times or confidence ratings to understand the judgment process. These additional measures can help, but they do not automatically identify the cause of a judgment. A fast answer might reflect strong knowledge, familiarity, or a quick guess. High confidence can coexist with error, while low confidence can accompany accurate caution. Such measures should be interpreted alongside, rather than substituted for, the primary veracity outcome.
Familiarity and repeated exposure
A false item does not arrive in a vacuum. Prior exposure can make it easier to process and can increase its perceived accuracy. In experiments on fake-news headlines, Pennycook, Cannon, and Rand found that earlier exposure raised later accuracy ratings of false headlines. The result supports treating repetition as a measurement condition, not proof that every repeated claim will be believed. It also complicates pre-post designs that reuse the same items. An improved or worsened score on a repeated test may partly reflect memory for the material rather than a change in general discernment.
Familiarity is not identical to source knowledge. A reader may recognise a sentence without remembering where it came from, or may remember an outlet without checking the specific claim. Asking “Have you seen this before?” helps describe exposure but is an imperfect retrospective measure. If researchers can control exposure experimentally, they can compare repeated and novel items. If they cannot, they should at least acknowledge the limitation and avoid treating all participants as encountering the same stimulus history.
Pretesting can itself create exposure. A pilot study, attention check, or teaching exercise that includes test items may influence later ratings. Interventions that explicitly discuss particular false claims can also make those claims familiar at follow-up. A defensible design uses fresh but comparable items where feasible, counterbalances item assignments, and reports whether participants previously encountered the claims. These design choices are especially important when a small apparent effect is being interpreted as broad resilience to misinformation.
Context changes the task
Judgments made in a quiet survey are not necessarily judgments made in a feed. A social-media-style display can add images, engagement counts, source names, and sharing prompts. In experiments on social-media context, researchers found that thinking about sharing while judging headlines interfered with truth discernment relative to an accuracy-focused context. This does not mean that every feed always impairs judgment. It shows that the instruction and surrounding activity can alter what a headline task elicits.
Time constraints are another context variable. Under time pressure, a participant may have less opportunity to inspect details or reflect on conflicting cues. Yet a group difference under a timed task should not be relabelled as a stable personal deficit. The signal detection study of time pressure is particularly instructive because it distinguished poorer discrimination from merely saying “true” more often. Researchers should record timing instructions, actual response times, device conditions, and whether external checking was permitted.
Political congruence, personal relevance, and emotional salience can shift the task’s meaning. A participant may know a claim is false but respond to what it signals about a group; another may have specialised knowledge that others lack. There is no single correction for all such effects. Balanced item sampling, pre-registered analyses, and transparent reporting make it easier to see which patterns survive changes in topic and context. When the goal is an applied intervention, testing in the likely use context is more informative than assuming a laboratory scale transfers unchanged.
How to interpret a score responsibly
A useful report states the instrument, item source, field dates, language, sample, response format, true/false composition, scoring rule, and uncertainty. It shows true-item acceptance and false-item acceptance separately. If it reports a discernment difference, it explains how that difference was calculated. If it reports signal detection indices, it explains what counts as a hit and false alarm and how extreme response rates were handled. It does not invite readers to infer a person’s character from a short score.
For an intervention study, the most informative comparison is often not “Did false belief decline?” alone, but “Did separation between true and false items improve without increasing rejection of true information?” An intervention that makes participants doubt everything can reduce false endorsements and still damage useful knowledge. The systematic review of news judgments explicitly draws attention to the combination of spotting false news and doubting true news. Both sides belong in an evaluation.
Reliability and validity have distinct roles. If the same construct is sampled with parallel items under similar conditions, scores should be sufficiently consistent for the proposed use. But consistency alone cannot show that a scale predicts real-world checking or sharing. Conversely, a short task may be valuable for group-level research even if it is unsuitable for decisions about individuals. The required precision depends on the claim being made and the harm of a wrong classification.
Researchers should also resist converting an average group effect into a story about every member of that group. Differences in education, age, ideology, or technology use can be confounded by topic familiarity and item selection. Measurement invariance, representative sampling, and replication across claim domains become essential when the purpose is comparison. A convenient online sample and a single set of contemporary headlines warrant a narrower conclusion.
What the evidence does not show
A headline test does not reveal a general, permanent “gullibility” essence. The reviewed evidence does not establish that a low score diagnoses a cognitive disorder, that a high score guarantees resistance in every domain, or that one intervention will protect a person across formats and future events. The MIST validation provides evidence for its defined news-discernment use, not a universal susceptibility meter. Experimental exposure and context effects demonstrate that measured performance can move with the task itself.
Nor can a low rate of false-item endorsement alone prove good discernment. It may result from indiscriminate scepticism. An increase in true-item acceptance alone may result from a liberal response threshold. A person may accurately judge a headline and still share it for reasons the accuracy question never measured. The evidence supports careful separation of these outcomes, not a single moral ranking of readers.
Published studies also do not remove the problem of changing truth status, culturally specific items, and information ecosystems that differ from a survey. A result in one language, country, topic, or platform should be carried to another only with new evidence. The responsible conclusion is conditional: under the specified items and conditions, a defined group showed a defined pattern of judgments. That phrasing is less dramatic but more useful for building better tests.
Using the measure to ask a better question
Imagine a library programme that teaches source checking. A weak evaluation gives participants the same false headlines before and after and celebrates a lower false-endorsement rate. A stronger evaluation uses comparable fresh true and false items, records familiarity, reports acceptance of each class, and asks whether discernment improves. It may also include a separate task for actual checking behaviour. The design cannot capture every real-world consequence, but it makes the causal and measurement claims more honest.
The practical aim is not to assign a permanent label to readers. It is to understand which conditions help people tell credible information from misleading information without teaching them to reject everything. Measuring susceptibility well means preserving the distinctions that ordinary language often hides: false belief versus truth discrimination, discrimination versus response bias, and test performance versus lived information practice. Those distinctions turn a score into a question researchers can actually investigate.
Research references
- Maertens, R., et al. (2024). The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment. Behavior Research Methods. View source
- Sultan, M., et al. (2022). Time pressure reduces misinformation discrimination ability but does not alter response bias. Scientific Reports. View source
- Sultan, M., et al. (2024). Susceptibility to online misinformation: A systematic meta-analysis of demographic and psychological factors. Proceedings of the National Academy of Sciences. View source
- Pfänder, J., & Altay, S. (2025). Spotting false news and doubting true news: A systematic review and meta-analysis of news judgements. Nature Human Behaviour. View source
- Pennycook, G., Cannon, T. D., & Rand, D. G. (2018). Prior exposure increases perceived accuracy of fake news. Journal of Experimental Psychology: General. View source
- Epstein, Z., et al. (2023). The social media context interferes with truth discernment. Science Advances. View source