In the 2002 movie Minority Report, Tom Cruise plays with a futuristic computer like a maestro. With a flick of his wrist, data moves from one screen to the next, and a panoramic view of a park spins like a toy top.
“But if it’s virtual reality, why should he only use two arms? Why should he even have a humanlike form?” asks Jeremy Bailenson, director of Stanford’s Virtual Human Interaction Lab.
Last week, Bailenson’s team released new research that shows a person can master a virtual third arm within just a few minutes. The study explores the concept of “homuncular flexibility”, wherein people learn to control bodies different from their own by changing how they perceive human movements.
“It opens interesting opportunities, such as with the design of video game avatars, but could also help war amputees who are in rehab,” says cognitive neuroscientist and media researcher Wijnand IJsselstein, who wasn’t involved with the research.
The project began in the 1980s as a brainchild of Jaron Lanier, the computer science pioneer who coined the term “virtual reality” as well as homuncular flexibility. Back then, he and a few friends built unnatural virtual bodies like a lobster with three arms along each side of its midriff. They would guide these extra limbs by subtle motions, like an elbow twist or bend of the knee.
But fast-forward two decades, and virtual reality experts remained the only ones who could conquer homuncular flexibility. So Lanier teamed with Bailenson in 2004 in order expand the homuncular flexibility experiment with regular human subjects.
“When we first tried this with Jaron, most people failed miserably. Only he and others who are gifted spatially and mathematically could power a third hand,” says Bailenson. “But we learned in this study that the key isn’t intellect, but better motion tracking.”
Their first experiment asked 53 undergrads to pop balloons by kicking and punching in a virtual classroom. Each person wore motion-tracking bracelets around their ankles and wrists, which controlled their avatar’s appendages, and a first-person virtual headset, so they could watch their movements.
One group had regular moving limbs, but the others controlled their hands with their feet and vice versa. When normally oriented, people preferred using their arms and could pop nearly 300 in 10 minutes, as you might expect. But the team found that the subjects adapted to the switched condition within the same timeframe, popping as many balloons when their feet controlled their avatar’s arms. People still preferred using their virtual hands for the popping, even though that required awkwardly standing on one foot like the Karate Kid.
“The switched conditions tell us that people will adapt, and they’ll do whatever is the most useful way to complete the task,” says Andrea Stevenson Won, a Stanford doctoral student who led the study, which was published in the Journal of Computer-Mediated Communication.
Yet the team wondered if people could do even better at a task with an unnatural human template than they could with their normal body structure. So in the second experiment, virtual users were asked to grab boxes from two rows. One group had two arms and had to step forward to nab objects from the second row. The other folks had a third arm sprout from their chest that moved forwards or sideways with a flick of each wrist. Within two and a half minutes, users learned how to snatch more boxes with three arms compared with two.
Within two and a half minutes, users learned how to snatch more boxes with three arms.
The implications span the virtual realm, but might make a difference in real life too. Homuncular flexibility could add an extra degree of difficulty to motion-based video games, like those on Nintendo Wii, or expand the capabilities of Microsoft’s new HoloLens. Or a Minority Report-like computer may arrive sooner rather than later. Japanese tech company NEC provided an academic grant for this project, says Bailenson, because they’re developing a consumer platform where people can physically move data in a digital cloud.
Meanwhile, Andrea Stevenson Won is moving her thesis research toward using homuncular flexibility for treating pain. Doctors have shown that transforming self-perception can help with pain conditions, such as with mirror therapy for amputees. After losing an appendage, many people experience “phantom pain” as if the limb still remains. Invisible fists remain clenched or a lost foot sears in agony. With mirror therapy, a person places their intact limb in front of a reflective surface and pretends that it is the lost appendage. By doing so, the patient mentally works through the pain, remapping brain networks that govern the sensation. Prior work shows that a person can use a physical mirror for this therapy or lifelike projections on a computer screen.
“But many pain conditions aside from amputation involve abnormal representations of the human body. We’re seeing if changing body appearance in a virtual space can help,” says Stevenson Won. Though not reported in this study, she has preliminary evidence that homuncular flexibility improves complex regional pain syndrome in children.
“It’s a promising starting point that also questions our notion about what is self,” says IJsselstein. “Suppose you’re a fanatic gamer and you have a long-term exposure to an altered body, does that reprogram your sensation of self?”
A real-life analogy is ice skating. When you hop off the ice and walk a few meters, your legs feel heavy. That’s motor reprogramming of human perception.
IJsselstein assumes that similar aftershocks might surface with long-term use of homuncular flexibility, but, as with ice skating, our brains will easily rebound once we’re back inside our real bodies.
