Aircraft carrier crews already use a set of hand gestures and
body positions to guide pilots around the deck. But with an increase
in unmanned planes, what if the crew could use those same
gestures to guide robotic aircraft?
A team of researchers at MIT -- Computer Science student Yale
Song, his advisor Randall Davis and Artificial Intelligence
Laboratory researcher David Demirdjian -- set out to answer that
They're developing a Kinect-like system (Microsoft's Xbox 360 peripheral wasn't available when the team started the
project) that can recognise body shapes and hand positions in 3D.
It uses a single stereo camera to track crew members, and
custom-made software to detect each gesture.
First, it captures a 3D image of the crew member, and removes
the background. Then, to estimate which posture the body is in, it
compares the person against a handful of skeleton-like models to
see which one fits best. Once it's got a good idea of the body
position, it also knows approximately where the hands are located.
It zeros in on these areas, and looks at the shape, position and
size of the hand
and wrist. Then it estimates which gesture is being used: maybe the
crew member has their palm open or their fist clenched or their
thumb pointing down.
The biggest challenge is that there's no time for the software
to wait until the crew member stops moving to begin its analysis.
An aircraft carrier deck is in constant motion, with new hand
gestures and body positions every few seconds. "We cannot just give
it thousands of [video] frames, because it will take forever," says
Song. Instead, it works on a series of short body-pose
sequences that are about 60 frames long (roughly three seconds of
video), and the sequences overlap each other. It also works on
probabilities rather than exact matches.
In tests, the algorithm correctly identified the gestures with
76 percent accuracy. Pretty impressive, but not good enough when
you're guiding multimillion-dollar drones on a tiny deck in the middle of the
ocean. But Song reckons he can increase the system's accuracy by
considering arm position and hand position separately.