Apple released a ‘spatial computer.’ Here’s what that means.

On Monday, Apple unveiled the Vision Pro—its long awaited augmented reality headset. Rather than selling it as a virtual reality headset or even as AR goggles, the company is calling the new device its “first spatial computer.” Apple CEO Tim Cook even tweeted: “Welcome to the era of spatial computing.” So let’s dig in and look at what on earth (or in the metaverse) spatial computing is. 

First, a few specs so you can understand what the Vision Pro is capable of doing. The new high-tech device resembles a pair of ski goggles and is packed with 12 cameras, five sensors, and two 4K displays. While it is capable of immersive VR experiences like gaming, it is mostly designed to be used for augmented or mixed reality. To achieve this, the cameras on the outside of the device will capture footage that’s displayed in real-time on the screens inside it. That’s how digital elements, like email apps, FaceTime video calls, and everything else can appear as if they’re floating in the physical environment surrounding you. Apps can be controlled with your gaze, hand gestures, and your voice—or with a keyboard and mouse. All this tech doesn’t come cheap: The Vision Pro will go on sale next year for $3,499. Now, on to what this all means for the future of computers. 

Although “spatial computing” has had a few different definitions, it is essentially an extension of virtual and augmented reality. A spatial computer is a device that enables the physical and digital to co-exist and, in some cases, interact. So, instead of only being able to use apps on your smartphone or laptop, spatial computers—like the Vision Pro—allow you to check your email anywhere you want in the world around you. Your email app can appear to be projected on a wall, floating off to your left, or attached to your wrist. Similarly, instead of having to rely on a keyboard or mouse, you can typically interact with a spatial computer using voice commands, your gaze, and physical movements, as if the objects you can see are real.

In announcing the Vision Pro, Apple’s marketing materials show how the company thinks spatial computing could fit into modern life. The examples the Cupertino company gives are pretty instructive, if nothing new. 

In what it calls “extraordinary new experiences”, Apple says the Vision Pro can provide “an infinite canvas for apps at work and at home.” In other words, instead of being limited by the screen real estate your smartphone or monitor provides, you can be more productive by having apps occupy the entire three-dimensional space in front of you. You aren’t stuck with the one or two monitors that fit on your desk, but can have apps fully surrounding you.

Similarly, if you want entertainment instead of productivity, the Vision Pro “can transform any space into a personal movie theater with a screen that feels 100 feet wide.” And unlike in a regular movie theater, you can even play video games on it. (There’s also Spatial Audio to provide the surround sound.)

None of these featured capabilities are particularly revolutionary, considering devices like Meta’s Quest line of VR headsets and Microsoft’s Hololens. However, Apple’s execution seems to be on another level. The hands-on feedback from the tech press suggest that Apple has solved some of the problems that have prevented VR and AR headsets from going mainstream. 

Firstly, the screens and cameras are high enough resolution that things work without being blurry. According to Nilay Patel for The Verge, “It was totally workable for reading text in Safari.” Similarly, the video passthrough just worked. “It appeared with zero latency and was sharp, crisp, and clear. I happily talked to others, walked around the room, and even took notes on my phone while wearing the headset—something I would never be able to do with something like the Meta Quest Pro.”

Second, because the Vision Pro will work as a VR screen for Macs as well as be able to run some iPhone and iPad apps from the start, it will likely launch as a far more capable general purpose spatial computer than anything that has come before. From day one, you will be able to check emails, make FaceTime calls, send iMessages, browse the web, fill in spreadsheets, and do huge numbers of your daily computing tasks—but all through a spatial interface. The price will be a big barrier, but it’s the first spatial computer that could see widespread adoption.

Of course, there is one big question mark hanging over the Vision Pro—and spatial computers in general. Is using AR goggles to do general computing tasks something anyone really wants? And is there enough of a productivity benefit that it’s worth wearing a large pair of goggles? 

While Apple appears to have got closer to making a mainstream spatial computer than any company has before, it’s still unclear if the Vision Pro is an expensive solution still trying to find a problem to solve.