Move over, autopilot: This AI can avoid other planes

In February, for the first time, a Black Hawk helicopter flew itself around with no humans on board. The self-flying military helo project involved both DARPA and Sikorsky, which makes the UH-60 helicopters. 

Meanwhile, in some places, companies like Zipline and Wing are delivering goods by drone. Other companies are working on electric air taxis to transport people or cargo, and of course normal air traffic—commercial flights out of big airports, general aviation airplanes zipping out of others—is flying around, too. Factor in helicopters, hot air balloons, and more, and there can be a lot going on up there.  

With all this busyness in the skies, researchers at Carnegie Mellon are working on an artificial intelligence pilot system that can carry out tasks like predicting what another aircraft might do, or keep an eye out for nearby planes using cameras on an aircraft. The idea is that an AI like this could help fly drones, assist a human pilot, or even someday fly a plane on its own. 

Right now, in a flight simulator, the AI is able to figure out what another aircraft is doing, or might do, and then figure out how to safely land the plane accordingly. Think of the way a driver behind the wheel of a car notices another vehicle approaching an intersection, and begins preemptively planning what to do if the other car were to run a stop sign, for example. 

In this case, the AI is looking out for another plane, not a car, of course. “It basically looks at their behavior for 10 seconds,” says Jay Patrikar, a doctoral student in the Robotics Institute at Carnegie Mellon University. “It tries to judge: ‘They are here. What are they potentially likely to do?’” 

In that sense, it’s like an AI that can play chess, says Patrikar, thinking about what its move would be in advance if its opponent were to take a certain action.

[Related: The Air Force plans to test an AI copilot on its cargo planes]

Artificial intelligence systems need data to learn from. In this case, the team is gathering data from two real-world airports, both of them in Pennsylvania. One has an air traffic control tower, and the other does not. Patrikar says that at those airports the data they hoover up includes visual information from cameras located on a hanger or near the taxiway, spoken communication from the radios, weather data, and more. “We record the entirety of it,” he says. The idea is for the AI to be able to learn cause and effect by paying attention to all this information. 

“It knows the causality of things,” he adds. That means that the AI could learn, for example, that “it was because of the weather that they [a pilot] decided to do this particular thing.” The training the AI received in these scenarios has helped it learn how to navigate a landing in simulation, Patrikar says. 

Plus, an AI bringing an aircraft in for a landing at a small, uncontrolled airport must both follow FAA rules as well as other norms when interacting with other planes, Patrikar points out. “One of the ways humans trust each other is with our shared understanding of rules—our social norms,” he says. People on a busy sidewalk might decide how to pass each other by moving to the right, for example, and rules like that apply in aviation that the AI pilot must follow. 

[Related: This company is retrofitting airplanes to fly on missions with no pilots]

Related work in the real world, not in simulation, has the team putting cameras on aircraft like a Cessna 172 or a hexacopter drone. Those cameras and the AI are able to spot other aircraft in the area, identify them, and figure out how far away they are with a greater than 90-percent accuracy rate at a distance of 700 meters (about 2,300 feet). This kind of tech could help a human pilot in a small plane visually spot other traffic in the area. “I would like to have that system on my plane,” says Patrikar, who has a private pilot license. After all, artificial intelligence doesn’t blink.

To be sure, the Carnegie Mellon researchers are not the only people exploring the new frontier of artificial intelligence that can fly, or help fly, aircraft. The Zipline drone company has been working on a way to use microphones on its drones to listen for other aircraft in the area and then take evasive action to avoid any potential collisions. And notably, a company called Merlin Labs has also developed a digital pilot that could take the place of a human copilot. As one example, it’s working with the Air Force on equipping C-130J cargo planes with their system, so instead of a human crew of two pilots, the aircraft could be flown by a single human paired with an artificial copilot.