Fighter pilots, meet your AR adversaries

A pilot has two general ways to train at the controls of a plane: in a real aircraft up in the sky, or down below in a simulator. A station on the ground can allow aviators to practice a tricky maneuver like refueling their bomber behind a tanker, and it’s less expensive than getting a real bird in the air. But of course, the reality of actually being in the sky, and experiencing punishing G-forces in a nimble fighter jet, is different from ground-based practice. 

A company called Red 6 has developed their own system that fuses those two training scenarios, by using augmented reality to visually simulate other aircraft on a fighter pilot’s visor while they are actually flying. The approach aims to fill a lacuna that the company’s CEO and co-founder, Daniel Robinson, says exists when carrying out training sorties in the sky.

“A lot of the work we do as combat pilots takes place beyond visual range—i.e., a range at which the sensors on the airplanes can see the threats, but the pilot is not capable of physically seeing them with their eyeballs,” Robinson, a former F-22 Raptor pilot, says. At around 10 nautical miles, a pilot could see an actual aircraft outside of their plane. But if those adversaries are virtual, and simulated only on radar, of course there’s nothing for the pilot to actually glimpse. “When they transition from the radar to look up, there’s nothing there.”

That’s the issue that Robinson wanted to tackle. “I wanted to figure out a way to put synthetic entities into the field of view of the pilots,” he says, “and have them look, feel, and behave, as if they were real airplanes.”

[Related: The US’s flagship fighter jet simulator is now multiplayer]

The solution involves using AR in the pilot’s field of vision, meaning they can see the real world outside the airplane—an important detail, to be sure—and also see images of synthetic aircraft that aren’t actually there. Those computer-generated images, he says, would be flown by either AI or a real pilot on the ground. 

The goal is to make it seem like a pilot flying a real aircraft in the sky is actually flying with, or against, simulated aircraft—because there’s no substitute for actually being up there, as opposed to in a simulator. “Strapping yourself to something that’s capable of pulling 9 Gs, you can run out of fuel, you can crash into the ground, [and] people are shooting at you,” Robinson says. “The cognitive load on the pilot’s brain is massively increased once you start doing this for real.” 

[Related: Refueling B-52s in the sky is hard, so the Air Force is trying VR simulators]

“If we accept that we must fly real airplanes to train,” he adds, “and expose ourselves to that environment, the question is, how do we do it efficiently?” His answer to that is getting pilots up in the sky, but using AR, AI, and computing power to create synthetic but visible aircraft to interact with them. 

Robinson outlines the ways in which the tech works. “I can fly an intercept on an augmented reality KC-46 tanker,” he says. “I can go join up on the wing, fly alongside it, drop in behind it, [and] I see the refueling boom.” In that respect, it’s a similar training scenario to a project the Air Force kicked off earlier this year to explore more accurate ways, using tech like AR, for B-52 bomber pilots on the ground to practice refueling their birds behind tankers. 

In the scenario Robinson describes, he could continue flying his real aircraft after joining up with the mock tanker, and then join a simulated wingman who’s flying a fighter like an F-22, and then the real aircraft and the artificial F-22 could then fight against an simulated enemy. If you were on the ground looking up at that complex scenario through binoculars, all you would see was one real aircraft, while the pilot in it would see two artificial ones outside the canopy. 

The company just recently signed a contract of some $70 million to install their system into a T-38, which is a two-seat training jet used by the likes of the US Air Force and NASA. They call it ATARS, for Airborne Tactical Augmented Reality System. Next up could be using ATARS on an aircraft like an F-16. 

Robinson says he expects to get ATARS up and running and off the ground in the T-38 in the next 12 to 18 months. “The first plan is to integrate into T-38, [and] validate that it’s safe,” he says; then comes bringing it to additional aircraft. “Can we create a network in which multiple military airplanes can connect and train together,” he wonders, “because if we do that, we’ve ushered in a fundamentally new paradigm in training.”