hand_touchscreen_1

Wave hand. Turn any surface into a touchscreen

CARNEGIE MELLON (US) — New technology makes it possible to create touch-based interfaces almost at will, with just the swipe of your hand.

The system goes beyond previous work that allowed a depth camera system such as Kinect to be combined with a projector to turn almost any surface into a touchscreen.

The WorldKit system enables someone to rub the arm of a sofa to “paint” a remote control for the TV or swipe a hand across an office door to post a calendar from which subsequent users can “pull down” an extended version. These ad hoc interfaces can be moved, modified, or deleted with similar gestures, making them highly personalized.


“Depth sensors are getting better and projectors just keep getting smaller. We envision an interactive ‘light bulb’—a miniaturized device that could be screwed into an ordinary light fixture and pointed or moved to wherever an interface is needed,” says doctoral student Robert Xiao. (Credit: Chris Harrison)

Researchers at Carnegie Mellon’s Human-Computer Interaction Institute (HCII) used a ceiling-mounted camera and projector to record room geometries, sense hand gestures, and project images on desired surfaces.

But Robert Xiao, an HCII doctoral student, says WorldKit doesn’t require such an elaborate installation. “Depth sensors are getting better and projectors just keep getting smaller. We envision an interactive ‘light bulb’—a miniaturized device that could be screwed into an ordinary light fixture and pointed or moved to wherever an interface is needed.”

The system doesn’t require prior calibration, automatically adjusting its sensing and image projection to the orientation of the chosen surface. Users can summon switches, message boards, indicator lights, and a variety of other interface designs from a menu. Ultimately, the WorldKit team anticipates that users will be able to custom design interfaces with gestures.

Xiao developed WorldKit with Scott Hudson, an HCII professor, and Chris Harrison, a PhD student. They will present their findings April 30 at CHI 2013, the Conference on Human Factors in Computing Systems in Paris.

“People have talked about creating smart environments, where sensors, displays, and computers are interwoven,” says Harrison, who will join the HCII faculty this summer.

“But usually, that doesn’t amount to much besides mounting a camera up on the ceiling. The room may be smart, but it has no outlet for that smartness. With WorldKit, we say forget touchscreens and go straight to projectors, which can make the room truly interactive.”

Though WorldKit now focuses on interacting with surfaces, the researchers anticipate future work may enable users to interact with the system in free space. Likewise, higher resolution depth cameras may someday enable the system to sense detailed finger gestures. In addition to gestures, the system also could be designed to respond to voice commands.

“We’re only just getting to the point where we’re considering the larger questions,” Harrison says, noting a multitude of applications in the home, office, hospitals, nursing homes, and schools have yet to be explored.

This work was sponsored in part by a Qualcomm Innovation Fellowship, a Microsoft PhD Fellowship, and grants from the Heinz College Center for the Future of Work, the Natural Sciences and Engineering Research Council of Canada, and the National Science Foundation.

Source: Carnegie Mellon University