Robo Brain Project Wants To Turn the Internet into a Robotic Hivemind

Robo Brain isn’t exactly feasting on the internet. That implies a level of choice, or agency. What’s happening to Robo Brain is closer to force-feeding, as researchers from four different universities regularly cram its cloud-based computational system with data collected from the internet. So far, it has digested roughly 120,000 YouTube videos, a million documents, and a billion images.

The goal is as direct as the project’s name—to create a centralized, always-online brain for robots to tap into. The more Robo Brain learns from the internet, the more direct lessons it can share with connected machines. How do you turn on a toaster? Robo Brain knows, and can share 3D images of the appliance and the relevant components. It can tell a robot what a coffee mug looks like, and how to carry it by the handle without dumping the contents. It can recognize when a human is watching a television by gauging relative positions, and advise against wandering between the two. Robo Brain looks at a chair or a stool, and knows that these are things that people sit on. It’s a system that understands context, and turns complex associations into direct commands for physical robots.

If none of those tasks seem especially impressive, or challenging, that’s because they aren’t—for humans. Robots, meanwhile, need all the cognitive help they can get. Even the most autonomous machines follow narrow bands of preset behaviors, with no real ability to adapt to new situations without extensive, and often costly training or reprogramming. For robots to become a ubiquitous social good, adaptation is crucial. An eldercare bot, for example, might need to understand not only how to retrieve a steaming bowl of soup from a microwave, but where to place it on the table, in relation to its owner. A simple request, for the cup with stripes on it, or a bigger spoon, might hopelessly stump the machine.

Robo Brain’s solution is a vast, and rapidly expanding flowchart. The system siloes everything that’s poured into it, focusing all of its efforts on determining when and how to take certain physical actions. Robo Brain means business, at all times. So spoons and bowls have intersecting nodes, but don’t expect Kant’s writings to ever show up in this vocational hivemind. This attention to action puts the project within the same general territory as RoboEarth, an ongoing non-profit effort to create a robot-readable storehouse of knowledge. The key difference, though, is that Robo Brain builds itself, adding to its network of conclusions and related actions (RoboEarth’s files have to be processes and organized by humans). It also differs from other “deep learning” approaches to artificial intelligence, such as Google Brain. Instead of trying to mimic the way humans perceive and process data, Robo Brain simply organizes information, grouping and connecting topics by context. Chairs are broken down into subsets of chairs, with branches leading to what chairs are for, and how they relate to other kinds of furniture. Coffee is connected to mugs, as well as to the motion-planning related to pouring liquid. It’s the brute force approach to cognition, dodging the nuances of actual intelligence, in order to serve up specific orders. Tell a robot to grab you a cup of coffee, and it can query Robo Brain, and then piece together a set of relevant commands.

But as the system’s sprawling graph continues to grow, Robo Brain is looking less like the illusion of intelligence, and more like the real thing. According to Ashutosh Saxena, assistant professor of computer science at Cornell University, the best example yet came when researchers asked one of the project’s three robots to make affogato. The bot, a two-armed, highly-dextrous PR2, queried the system, and discovered that affogato was an italian dessert composed of ice cream and coffee. Without any human nudging or intervention, the robot located the coffee, figured out how to get it out of a dispenser, and poured it over the scooped ice cream.

Again, this is child’s play for us big-brained biologicals. For a robot, the ability to respond to a relatively vague request with a chain of queries and commands which add up to a specific, and correct result, is astonishing. And to be clear, it’s not as though researchers fed a complete affogato or coffee-preparation sequence into that PR2. The data pumped into Robo Brain is curated by topic, but raw in nature—DIY videos, instruction manuals and how-to documents. The solutions output by the system were just as seemingly off-the-cuff. “Right now it’s automated in the sense that the four universities, we push the knowledge into system,” says Saxena. “We don’t have to do anything else.”

At the moment, Robo Brain’s input comes from researchers at Brown, Cornell, Stanford and UC Berkeley. By October, though, the project will expand to 10 universities in total, and by the end of the year applicants from any school will be able to apply for access. At that time, Saxena expects Robo Brain to have absorbed a billion videos and 10 billion documents, with cloud computing costs hitting $1000 per day. Current funding, some of which comes from sponsors like Google and Microsoft, is at roughly $1.5M, enough to keep the system well-fed for years. Even so, Saxena estimates that it could take a half-decade before the system is truly fit for duty, and ready to make a difference.

“We did it completely in open source. Unlike some other university programs, or some companies, our goal is not to make a profit,” says Saxena, who sees eldercare and general household bots as Robo Brain’s primary initial beneficiaries. “We’re making the world a better place.”