Share this story
Close X
Switch to Desktop Site

Kinect hack makes Gmail's April Fools' prank real

A new Kinect hack lets computers understand American Sign Language, and control Gmail.

View video

Kinect, a motion-sensing peripheral for the Microsoft Xbox 360, has been named the fastest selling consumer electronics device of all time.


View photo

Soon, you may not necessarily need to be fluent in American Sign Language (ASL) in order to interpret it. Scientists have hacked Microsoft's Xbox Kinect motion control sensor to read ASL.

The real-world feat is reminiscent of Google’s April Fools’ Day prank this year, in which the company falsely debuted a featured called Gmail Motion that allowed users to translate bodily gestures into words and email commands.

About these ads

RELATED: The 10 weirdest uses for a smartphone

The Kinect, which debuted in November, offered a revolutionary way to interact with computers without pushing any buttons or holding any device whatsoever, using only body motions to control Microsoft's Xbox game console. The add-on, which is essentially a motion-sensing webcam, uses an infrared scanner to create 3-D models of people as they move, allowing users to play games by swimming their arms, shimmying their bodies or performing other so-called natural interactions. The Kinect has proven very popular, with 8 million sensors sold worldwide within 60 days of its launch.

The Kinect drew the attention not just of gamers but of programmers as well, with a thriving community of hackers now testing the limits of what the sensor can be used for, such as helping mobile robots respond to gestural commands.

Now researchers at Georgia Tech are pairing up the Kinect device with custom software that can interpret a very limited American Sign Language vocabulary with greater than 98 percent accuracy.

The scientists initially only used a limited vocabulary of six signs — those for "alligator," "spider," "box," "wall," "behind" and "in" — all signs that involve broad gestures with the arms and body.

"What we're doing now is working on computer vision algorithms to get more information on hand shapes from the Kinect," said researcher Helene Brashear, a computer scientist and president of Georgia Tech spinoff company Tin Min Labs in Austin, Texas. In the future, the scientists also hope that an improved Kinect sensor with even higher-resolution imaging comes out.

The scientists are working on a game called Copycat aimed at helping deaf children practice sign language. "Ninety-five percent of deaf children are born to hearing parents, very few of whom are fluent in sign, so we want to support these children as much as possible," Brashear explained.

About these ads

In the long run, sensors like the Kinect could lead to ways for computers to understand sign language and translate it to English or other languages. "That's far off, but it could happen," Brashear told TechNewsDaily. "Right now, advances in technology have really helped the deaf community — video chat is huge there, for instance.

RELATED: The 10 weirdest uses for a smartphone

Follow Stories Like This
Get the Monitor stories you care about delivered to your inbox.