In Head Trip, PopSci explores the relationship between our brains, our senses, and the strange things that happen in between.
MANY ILLUSIONS are products of mismatched sensory inputs, evoked when one sense contradicts another. (It looks like just a noisy fan, but it sounds as if it’s speaking.) One of the most startling—and easiest to try by yourself—of these illusions is the McGurk effect, an audiovisual illusion first described by Scottish psychologist Harry McGurk and his assistant John MacDonald in 1976.
If you search YouTube, you’ll find many videos of the McGurk effect. You watch a person’s face as they speak a single syllable—usually ba—over and over again. After a while, the person will start mouthing a different syllable, usually fa. Many listeners will “hear” the accompanying audio changing to match this. In reality, though, it does nothing of the sort; the sound being played remains the same throughout. “Ba! Ba! Ba!”
So what’s going on? Michael Beauchamp, a professor of neurosurgery at the University of Pennsylvania, has spent much of his career investigating the McGurk effect. “It’s what I think about all day, every day,” he laughs. In a 2012 paper in the journal NeuroImage, Beauchamp and colleague Audrey Nath examined the link between the effect and neural activity in a region of the brain called the left superior temporal sulcus (STS).
This STS forms a physical bridge between the visual cortex and the auditory processing region (a fact Beauchamp demonstrates with a 3D printout of his own brain). One of this brain region’s many important functions is processing multisensory audiovisual input. “[The STS] puts auditory information and visual information together,” Beauchamp explains. “That’s why we think it’s important for the McGurk.”
The 2012 research examined functional MRI data to study left STS activity in people who experienced the McGurk effect and to compare it to left STS activity in those who didn’t. There were, indeed, increased levels of activity in the first group. However, Beauchamp makes sure to caution that the results don’t constitute anything as definitive as “the STS causes the McGurk effect,” given the inherent complexity of the brain. “I wouldn’t feel comfortable [being that definitive] without a much larger sample size,” he says.
Still, the study did hint at one important fact that has been the focus of much subsequent research. “Some people always get the McGurk effect, and some people never do,” Beauchamp says. “[But] there’s also a whole spectrum in the middle. We are super certain of this; we’ve seen [it] in hundreds of people.”
The existence of this spectrum suggests that the effect—and thus the interaction between vision and hearing in multisensory processing—is more complex and nuanced than many scientists once believed. (This includes McGurk himself, who claimed that 98% of people always experience the full effect, while the remaining 2% never experience it at all.)
It also suggests the whole concept of “illusion” is worth re-examining. We tend to assume that experiencing an illusion constitutes a failure of our senses—that we’ve been fooled, and that in the process, we’re coming up against the limits of our brains’ ability to make sense of the outside world. But Beauchamp’s study proposed that the real picture might be more subtle: “We speculate that McGurk perceivers have more liberal criteria for integrating auditory and visual speech information. Even if the auditory and visual information is mismatched, McGurk perceivers integrate it. This might provide an advantage under conditions of high levels of auditory or visual noise, at the cost of being misled by McGurk stimuli.”
That means, in some cases at least, the susceptibility to illusions may be adaptive, rather than maladaptive, because illusions are ultimately induced by the brain doing its best to make sense of mismatched or contradictory sensory information. This also raises the question of how our neurological centers might adapt to a change in the quality of that information. (As someone who has acquired hearing loss—I have damage to the cochlea in one ear, the legacy of a stray elbow in a childhood basketball game—I find this idea has personal resonance.) So do we know how, or if, the STS and the rest of the brain adapt to a long-term change in the reliability of one of the senses?
“It’s a fascinating question—and an open one. We know the brain is plastic,” he says, adding that finding ways to use this plasticity is one of the goals of his team’s research. “For example, a lot of people’s hearing declines a lot faster than their vision does, so if we could help them to become more attuned to visual information, that might help [compensate for] hearing loss.”
The extent of the brain’s plasticity in this respect is underlined by one more remarkable detail that Beauchamp’s research has uncovered: The McGurk effect can be permanent. “If you watch the same McGurk effect clip for a long time, you’ll get the illusion even if you’re not looking at the screen. Basically, your brain is getting rewired; you don’t even need to see the face anymore, because your brain has been convinced, ‘OK, the auditory part is wrong, so I’ll go with what the visual part is saying,’” the professor explains.
Again, you can try it yourself: “Go on YouTube,” Beauchamp says. “Watch [one of those] videos for a minute a day for a few days, and then listen to it again without looking. My prediction is that you’ll still get the McGurk effect.”
Read more PopSci+ stories.