Why US intelligence wants a new way to make virtual, 3D models

On July 12, the Intelligence Advanced Research Projects Activity (IARPA) announced that it wants a new way to make photorealistic virtual models. The organization’s mission is researching and developing new tools for intelligence agencies, like the CIA and the FBI, as well as others through the US government and the Department of Defense. Intelligence is the profession of finding useful, actionable information, and the new project on virtual renderings is a way to ensure that when people on the ground are sent to a building they’ve never visited before, they can find all the right side doors in.

Spy thrillers make it seem like government agencies have access to perfect information about the world, from the panopticon of 1998’s Enemy of the State through the superhumanly perceptive agencies of the 2000’s Bourne trilogy to the all-knowing and all-powerful AI “Entity” of 2023’s Mission: Impossible. Intelligence agencies guard knowledge of what they can and cannot do so as to not dispel that notion. This request, for a tool to create useful, 3D virtual models, suggests that movie scenes where an agent enhances a camera view until it’s a perfect life-size picture remain the stuff of fiction.

What IARPA wants help with, in brief, is the ability to give people, like soldiers or first responders, an explorable 3D map of a place made from real imagery, rather than a 2D depiction of the place.

The organization calls the initiative WRIVA. “The Walk-through Rendering from Images of Varying Altitude (WRIVA) program seeks to produce innovations that will advance 3-D site modelling capabilities far beyond today’s state of the art, giving personnel virtual ‘ground truth’ with unrivaled insights into locations that would be difficult, if not impossible, to view,” reads the announcement from the Office of the Director of National Intelligence.

Modeling in this instance conjures to mind specific renderings of locations made by computer software, which is the intent, but it’s worth considering how recently the models procured by the CIA were literal, physical models, with parts that might accompany an electric train set or a hobbyist wargame. 

“In support of the raid that resulted in the death of Usama Bin Ladin, National Geospatial Intelligence Agency (NGA) modelers built Abbottabad Compound 1 Model,” notes the CIA’s description of the 1:84 scale model. Before Navy SEAL Team 6 went on the May 2011 raid, they used this model, where 1 inch matches 7 feet of the compound, to understand the compound and its surroundings. The CIA continues, “This model was used to brief President Obama, who approved the raid on the compound.”

The compound was under surveillance for a long time, and had the virtue of housing an occupant who was unlikely to leave. That allowed surveillance images to be collected for building the model, to ensure the SEALs found the highest profile target in the War on Terror. Not content with merely a miniature model, the SEALs also rehearsed the raid in a life-size mock-up of the compound in North Carolina.

WRIVA wants to offer that kind of detail and clarity, without the painstaking work of physical modeling, thus expanding who gets access to such walkthroughs.

“Imagine if the Intelligence Community (IC), law enforcement, first responders, military, and aid workers could virtually drop into a location and familiarize themselves before their feet even hit the ground,” reads the announcement.

In cases like the Abbottabad raid, where the mission was specifically about sending armed special operations forces into danger, knowing the external layout of a building and its surroundings allowed the raiders to move through the exterior parts of the compound with some familiarity. In rescue work, being able to pull up the outside of a building could give first responders en route a way to search for entrances and features familiar to locals but unknown to new arrivals.

DARPA, which tackles blue sky technological development for the military and is a type of cousin to IARPA, has explored development in a similar lane with its subterranean challenge. In this competition, competitors built robots that could go inside and map out buildings, creating useful tools for any humans that follow. This has immediate implications for rescue work, and also can be easily adapted to military use, where a robot explores a dangerous cave possibly filled by armed enemies, before any soldier is put at risk.

With IARPA’s project, it’s the observable outsides of buildings that become fodder for virtual model making. The announcement says the goal is to make “photorealistic virtual models using satellite, ground-level, and other available imagery.” (While IAPRA did not mention artificial intelligence in its announcement, the companies named as leads include Blue Halo and Raytheon, which have experience working with AI, which could be one way to tackle this problem.) The trio of satellite, ground level, and other available imagery sounds a lot like the methods used by open-source analysts to try and identify the location of videos and events in publicly available photography. With access to the resources on hand across the US intelligence community, what can be done in open source should be seen as just the beginning, not the end state, of what IARPA is asking companies to do.