A face on a screen shapes its lips for one syllable while a loudspeaker plays another. A listener who hears the soundtrack alone identifies it correctly. With the face in view, the same soundtrack can seem to say a third syllable. Closing the eyes changes what is heard. The experience is vivid because the sound has not been replaced; its interpretation has changed in the presence of visual speech.
Harry McGurk and John MacDonald reported this audiovisual speech illusion in their 1976 paper “Hearing lips and seeing voices.” In their classic example, an acoustic ba was paired with a visible ga, and adults often reported da. The finding established that seeing a speaker’s mouth can influence what a listener reports hearing, even when the visual and auditory signals conflict. It did not establish that all speech perception works like an artificial mismatch, or that a single illusion score captures someone’s everyday communication ability.
How an accidental mismatch became an experiment
The original paper used dubbed video of a speaking face. Its force came from a controlled conflict: the visible articulatory movement and the audible syllable specified different speech gestures. Listeners were not merely asked whether the picture matched the sound. They reported what syllable they perceived. When an incompatible visual gesture changed that report, the result challenged any account in which speech perception is always computed from sound alone and vision contributes only later commentary.
MacDonald’s later account of the discovery describes how the stimulus was constructed and how the unexpected percept emerged during work on audiovisual speech. That history matters because the striking classroom version can make the effect appear inevitable. It took a particular recording, a particular dubbing and a carefully observed response to reveal it. The paper was an experiment about a defined audiovisual conflict, not a claim that mouths always overrule ears.
Different mismatches can produce different reports. Sometimes listeners report a fused category that is not identical to either input; sometimes reports resemble one component or include a combination. The exact syllables, acoustic clarity, visual articulation and response options matter. A video that elicits da from one audience may elicit ba from another. That variation is not necessarily a failure to pay attention. It is part of the phenomenon to be explained.
What does the listener actually integrate?
Speech is an action that produces both sound and visible movement. Lip closure, jaw opening and the timing of those movements provide information about what a speaker may be saying. A listener’s brain can use that information alongside acoustics, especially when sound is uncertain. The McGurk stimulus deliberately makes the sources disagree, so the resulting percept shows that the streams interact. It does not by itself specify a unique neural computation.
One broad interpretation is that the listener estimates which auditory and visual signals belong to the same speaking event and then combines information when they appear to share a cause. Magnotti and Beauchamp developed a causal-inference model that accounts for responses to McGurk and other incongruent audiovisual speech under tested conditions. It is a model comparison, not a final proof of how each person hears all speech. Other mechanisms and levels of processing remain under investigation.
Timing helps determine whether the two signals seem to belong together, but audiovisual speech is not a metronome with zero tolerance. Natural lips can move before the associated acoustic event; recording and display systems introduce delays. An experiment should state its timing and test whether a response persists when synchrony changes. Merely saying that visual and auditory information “merge” leaves out when and why the system should treat them as one event.
The syllable report is an outcome of perception and decision. A listener may have a continuous impression but must select a label from available answers. Forced-choice menus can constrain what is reportable. Open responses can capture unexpected syllables but require coding decisions. Therefore, a percentage of da reports is partly a property of the stimulus and task, not an unmediated readout of sensory fusion.
Individual differences and stable variation
Basu Mallick, Magnotti and Beauchamp studied variability across people, stimuli, time and response format. They found substantial variation in how often the illusion was reported, yet individual response patterns showed notable stability over a one-year retest under their studied procedures. This combination is instructive. A person may respond differently from a neighbor without being random, but the measured response still depends on the particular stimulus and task.
The same authors tested new stimuli and different response formats, again finding that design choices affected rates. A single edited clip cannot support a strong trait claim such as “this person is a visual learner” or “this person has poor auditory processing.” Even a stable response to one set of clips would need careful interpretation. The result concerns how an individual handles those conflicts, not a comprehensive profile of speech understanding.
Nath and Beauchamp examined neural correlates of interindividual differences in the McGurk effect. Their work links variation in behavioral responses with activity and connectivity in regions involved in audiovisual speech processing. Such findings can help locate candidate processes and guide new tests. They cannot reduce a listener’s communication life to one brain area or turn a laboratory response into a diagnosis.
Experience, language, hearing conditions, attention and the stimulus itself can all be relevant, but each requires an appropriate design to test. The original 1976 demonstration is not an automatic explanation of a child’s speech development, a bilingual listener’s experience or an older adult’s conversation in noise. Those are separate research questions with their own populations and outcomes.
Why a mismatch is not ordinary conversation
In most conversations, visible and audible speech come from the same speaker and broadly agree. Faces also convey turn-taking, emotion and social attention. The McGurk stimulus strips away much of that context and creates a conflict that a speaker would not normally produce. It is powerful precisely because the mismatch makes the contribution of visual speech hard to ignore. Its exceptional design limits the inference to everyday communication.
Alsius, Paré and Munhall’s critical review warns that processing an incongruent McGurk stimulus differs from processing matching audiovisual speech, both in experience and in studied neural responses. They advise caution in using illusion magnitude as a proxy for all audiovisual integration. A measure can be very good at revealing one process while being incomplete for another. The better question is what the task isolates and what it leaves out.
For example, seeing a speaker’s lips may improve comprehension when speech is noisy, but the benefit of congruent visual speech should be tested directly. A high or low fusion rate with an artificial ba/ga mismatch does not automatically predict how well someone understands a friend in a busy café. Real speech contains words, context and many cues beyond one consonant-vowel syllable. A transfer claim requires a transfer study.
The distinction also matters for technology. A video call with delayed sound may feel disorienting. But a McGurk demonstration cannot set a universal acceptable latency for all calls. A product test should use representative speech, devices and users and measure comprehension and comfort under realistic timing. The illusion supplies a reason to pay attention to audiovisual alignment, not a finished engineering standard.
From a perceptual report to a mechanism
The report “I heard da” is a result to explain. It does not tell us at which stage of processing visual information changed the outcome. Possibilities include early interactions among sensory representations, categorical speech decisions, expectations about articulatory gestures and task-specific response selection. Research can compare predictions by varying auditory clarity, visual reliability, timing and instructions.
A strong mechanism claim should survive more than one clip. It should explain why some incongruent pairings create fusion, why others do not, and why people vary. Magnotti and Beauchamp’s causal-inference account is valuable because it puts those questions into a model with testable predictions. Still, fitting a model to observed choices is not the same as observing the brain’s computation directly. Competing models can sometimes explain similar response patterns.
Neural evidence can add constraints, as in Nath and Beauchamp’s work, but it also needs careful interpretation. Activity associated with a percept may be part of producing it, evaluating it or responding to it. To distinguish these, researchers need control conditions, timing information and interventions or converging measures. A colorful activation map is not a substitute for a behavioral explanation.
How to read a demonstration responsibly
If a reader tries a McGurk video, first listen without looking, then watch and listen, and finally compare with someone else. The variation itself is educational. Check whether the recording quality, earphones, screen size or playback delay changes the result. A classroom exercise can make the role of visual speech tangible, but it should not grade students on whether they report the expected syllable.
In research, the procedure should identify the talker and stimuli, record synchronization, describe response choices and report responses to audio-only and visual-only conditions where appropriate. A study claiming an effect in a population should show the distribution across participants, not just an average. A study comparing groups should ask whether the stimuli were equally clear and the task equally accessible to both.
Replication should preserve the important features of the original task while also testing generality. If a result depends on a single talker, it may reveal something about that stimulus rather than a general property of the listener. If it survives across talkers, syllables and response formats, the claim can grow. The strength of an illusion is not that it always happens; it is that its conditions can be varied to learn how perception works.
For a study of hearing difficulty, the appropriate comparison is especially important. Researchers would need to measure hearing and visual access directly, decide whether sound levels are comfortable and report whether participants can see the mouth clearly. They should include an outcome related to speech understanding, not only the frequency of a fusion syllable. A group difference on one edited clip might reflect audibility, visual clarity, language experience or answer format. It cannot be interpreted as a single deficit in “integration” without ruling out those alternatives.
Researchers should also distinguish a listener’s report from a demand to conform. If a demonstration is introduced as “everyone hears da,” some participants may doubt a different experience or choose the expected answer. Neutral instructions and individual reporting make variation more informative. A perceptual effect can be both real and sensitive to the way the experiment is presented; honest methods take both possibilities seriously.
Institute synthesis: support the whole act of communication
The Institute uses this paper to illustrate a principle about human capacities: sensory channels work together within a meaningful activity. A person in conversation uses eyes, ears, expectation, context and relationship. The 1976 mismatch isolates one part of that integration. It reminds designers and educators that visual access can matter, while warning them not to reduce communication to a single multisensory trick.
A practical response to that lesson might be clear sightlines, captions and good audio, chosen according to what people need. Those are design proposals, not outcomes tested by McGurk and MacDonald. Captions add linguistic information rather than replicating lip movements. Improving one channel may reduce dependence on another. A system should be evaluated with the people and tasks it is meant to serve.
Communication can also involve a choice not to use visible speech: a person may have privacy needs, visual fatigue or a device that does not show faces well. Respecting agency means offering accessible alternatives and checking comprehension, not insisting that everyone watch a mouth. The experiment is most useful when it widens the questions asked about communication rather than turning one perceptual response into a norm.
The lesson extends to meetings and classrooms where sound quality is poor. A teacher may face the room while speaking, or a microphone may introduce distortion. Better audiovisual conditions could help, but the McGurk effect alone does not quantify the benefit of any particular change. An evaluation can ask students what they understood, compare captions and audio quality, and test with actual lessons. The illusion points to a design variable; representative tasks determine whether a proposed change helps.
What the evidence does not show
The 1976 paper does not show that everyone hears da from every ba/ga clip. Later work documents large variation across people and stimuli, as well as effects of response format. Nor does the effect show that vision always dominates hearing. It demonstrates that visible speech can alter a report about an incompatible sound under particular conditions.
An illusion rate is not a complete measure of audiovisual speech integration. Alsius and colleagues distinguish the processing of conflict stimuli from congruent communication. A laboratory percept does not establish a person’s language ability, clinical status or likely benefit from a video call. Neural correlates and computational models constrain theories, but neither supplies a final, unique mechanism on its own.
The paper also does not justify a universal rule for classrooms, clinical practice or interface design. Those applications require direct tests with relevant users, tasks and outcomes. This entry’s suggestions about access and design are Institute synthesis. The research-supported conclusion remains bounded: what we see a speaker do can change what we hear them say.
Return to the face on the screen
The listener opens their eyes again. The same loudspeaker still plays ba, but the visible lips can make the sound seem different. The experience is neither a failure of hearing nor evidence that the listener is easy to persuade. It is a window onto a perceptual system that builds a coherent event from more than one source of information.
McGurk and MacDonald’s small paper changed how speech perception could be questioned. Its legacy is strongest when the original contrast stays clear: a controlled conflict reveals visual influence on an auditory report, while the richer work of understanding ordinary speech continues beyond the illusion.
Research references
- McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748. View source
- MacDonald, J. (2018). Hearing lips and seeing voices: The origins and development of the McGurk effect and reflections on audio-visual speech perception over the last 40 years. Multisensory Research, 31(1–2), 7–18. View source
- Alsius, A., Paré, M., & Munhall, K. G. (2018). Forty years after hearing lips and seeing voices: The McGurk effect revisited. Multisensory Research, 31(1–2), 111–144. View source
- Basu Mallick, D., Magnotti, J. F., & Beauchamp, M. S. (2015). Variability and stability in the McGurk effect: Contributions of participants, stimuli, time, and response type. Psychonomic Bulletin & Review, 22, 1299–1307. View source
- Nath, A. R., & Beauchamp, M. S. (2012). A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. NeuroImage, 59(1), 781–787. View source
- Magnotti, J. F., & Beauchamp, M. S. (2017). A causal inference model explains perception of the McGurk effect and other incongruent audiovisual speech. PLOS Computational Biology, 13(2), e1005229. View source