Video: Aircraft Carrier Crews Guide In Robot Planes With Visible Hand Gestures

By Rebecca Boyle Posted 03.14.2012 at 11:06 am 10 Comments

Carrier Landing An aircraft director signals to a pilot after a safe landing aboard the aircraft carrier USS Carl Vinson. US Navy via Wikimedia

Landing airplanes on moving ships is no mean feat, but this will be especially true when the airplanes are unmanned. Along with making decisions, autonomous airplanes will have to heed their human counterparts during aircraft carrier takeoff and landing — but can a robot read and understand arm-waving signals?

The problem is complicated in at least two ways — first, the airplane must determine whether the human’s hands are up or down, elbows in our out. Second, it has to red which gesture the human is making, and what it means. MIT PhD student Yale Song is trying to solve these problems.

Song and fellow scientists recorded various people performing a set of 24 gestures aircraft carrier deck personnel use, including arm waving and folding, and hand movements. They built software that determined each person’s elbow, wrist and hand positions, including whether palms were open and whether thumbs were up or down. They completed that portion of research last year. Then, the team had to classify all these gestures according to their meanings. But this is complicated, because deck signals are a complex ballet of movement — it’s not like a seaman makes one motion and then stops for a beat before starting another. So the algorithm has to determine a gesture’s meaning without a clear beginning, middle and end.

Aircraft Hand Signals: Melanie Gonick/MIT

The team accomplished this by breaking down the videos into clusters of frames, in which the analysis overlaps. As MIT News puts it: “The second sequence might start at, say, frame 10 of the first sequence, the third sequence at frame 10 of the second, and so on.” This way, the algorithm can calculate the probability that a given sequence belongs to one of the 24 gestures in the catalog. The researchers tested it and the algorithm correctly identified the hand signal sequences 76 percent of the time.

Song believes this can be improved by tweaking the computations, according to MIT. The work appears in the journal ACM Transactions on Interactive Intelligent Systems.

[MIT News]

Previous Article: Ocean-Crossing Robots Reach Hawaii, Setting a New Distance Record

Next Article: First Humanoid Robot in Space Inspires a Grip-Enhancing Robotic Glove For Humans

10 Comments

Link to this comment

david_forbus

03/14/12 at 11:18 am

Why is this necessary? Why can't the people coordinating things just drive it with a joystick?

Link to this comment

BillyBlackhawk

03/14/12 at 11:52 am

This is designed to simplify the process. Using hand signals that have already been developed and having each have specific purpose and meaning, why would you want to have to use a joystick and issue the commands in a different format?

Furthermore, I believe they are taking a step forward and looking to the future. If reliable software can be created for this application, what is to stop it from being used in others. Whether or not this will catch on or will succeed with such efficiency is yet to be seen.

Link to this comment

Jivaii

03/14/12 at 1:30 pm

The hand signals are there because the pilot in the aircraft isn't going to be watching everything happening on the flight deck, and trust me, it can change at a second's notice.

The pilot is watching the PC (Plane Captain) for commands on where to move the aircraft, how fast to move, whether the wings need to be folded or not, if he should drop hook or do a flight controls test and so many other things. Even something like turning on the engine, since the pilot can't see under the aircraft

If the UAV/robots can take those same commands, it means that the PCs don't need to relearn all their command signs, and the UAV can integrate into the flight deck's movements smoothly.

Proud Sailor of the USN

Link to this comment

Jivaii

03/14/12 at 1:35 pm

Alas, the lack of an edit button bites me again. I would like to see how the system would handle a hand off from on PC/Director to another.

Proud Sailor of the USN

Link to this comment

extremechiton

03/14/12 at 1:59 pm

wouldnt kinect be easier to work with?

Link to this comment

TheArtist

03/14/12 at 3:53 pm

Sounds very complicated, imagine if a strong gust of wind blows and the Aircraft Conductor needs to lurch to brace himself, sometimes these guys drop to a knee, would that cause the plane to lurch along with his movement? Vice versa with the plane movement. Pretty complex!

Link to this comment

contra_dictions

03/14/12 at 9:21 pm

Innovation with Nintendo Wii technology. Just be sure not to sneeze.

Link to this comment

david_forbus

03/15/12 at 12:56 am

A computer is not a human being. Tasks that are very simple for us are very difficult for a computer. Especially if the possibility exists for ambiguity. Eventually a computer could be created to handle such, but I think it would be easier for humans to adapt to the robots and not the other way around. There will be unforeseen situations where the computer will misinterpret the signals or situation. Computer don't have common sense and every possibility has to be programmed in. That is difficult to predict.

Link to this comment

loco

03/15/12 at 7:47 am

a buck says this dude's movin to china after he's finished with his degree...

Link to this comment

alexdhenley

03/16/12 at 2:47 pm

so any hacker with a Kinect (read: anyone that makes the Kinect worthit) will be able to control these UAVs? great. as if it wasn't scary enough that we already had drones in our skies.