Facebook Is Retooling 360 Video To Use Less Data
"We put a sphere inside of a cube"
“You’re going to start seeing a lot more 360 degree video on your News Feed,” Facebook CTO Mike Schroepfer told Popular Science last week at the Future of AI conference at NYU.
And he was right.
Today, Mark Zuckerberg and Facebook’s engineers released information about how they’re making 360 degree video up to 80 percent smaller than the original file, to make streaming the content easier on computers.
The answer lies in geometry. Facebook’s engineers explain that instead of thinking about 360 degree video as a sphere, it’s better to think of the final product as a cube, a shape with 6 equal sides. That equality means less inherent distortion than a sphere, which is geometrically dissimilar than the sensor the image is being captured on. This is great for viewing on a flat screen, but not in virtual reality.
For VR, videos uploaded to Facebook start as a sphere. But Facebook’s software deals with spherical video by putting it into a pyramid, which as a tetrahedron can represent 3 dimensions. Then the software “unfolds” the pyramid, and crops it, making it into one part of the cube. This is inherently a visual process, so let this GIF make it easier to understand.
Facebook makes 30 viewports, or angles to view, in 5 different resolutions, meaning there’s 150 versions of each video on Facebook’s servers that are dynamically streamed to a viewer in virtual reality. To manage data, Facebook splits video in one second segments that are streamed based on the orientation of the user’s head.
They’re hoping on using machine learning to make this process smarter in the future, but for now are relying on hand-coded software.
Schroepfer did say that 360 degree video will still require wi-fi, however, and we won’t see streaming that makes sense over cellular towers in the near future.
Encoding for 360 Video and VR
Both 360 video and VR provide immersive viewing experiences, but this richness comes with a set of new and complex engineering challenges. The large file sizes can be an impediment to delivering video quickly in a quality manner at scale, and the videos can appear distorted during playback. We solved these challenges with new software that transforms 360 video from the standard equirectangular format into either cube maps or pyramid layouts, which reduces the file size without sacrificing quality or resolution.Posted by Facebook Engineering on Thursday, January 21, 2016