OmniTouch hand_1

Turn any surface (skin!) into touchscreen

CARNEGIE MELLON (US) — A wearable projection system turns paper, walls, and even skin into an iPad-like touchscreen.

OmniTouch employs a depth-sensing camera, similar to the Microsoft Kinect, to track the user’s fingers on everyday surfaces. This allows users to control interactive applications by tapping or dragging their fingers, much as they would with touchscreens found on smartphones or tablet computers.

The projector can superimpose keyboards, keypads, and other controls onto any surface, automatically adjusting for the surface’s shape and orientation to minimize distortion of the projected images.


“It’s conceivable that anything you can do on today’s mobile devices, you will be able to do on your hand using OmniTouch,” says Chris Harrison, a Ph.D. student at Carnegie Mellon University.

The palm of the hand could be used as a phone keypad, or as a tablet for jotting down brief notes. Maps projected onto a wall could be panned and zoomed with the same finger motions that work with a conventional multitouch screen.

Harrison was an intern at Microsoft Research when he developed OmniTouch in collaboration with Microsoft Research’s Hrvoje Benko and Andrew D. Wilson. Harrison described the technology at the Association for Computing Machinery’s Symposium on User Interface Software and Technology (UIST) in Santa Barbara, Calif.

The OmniTouch device includes a short-range depth camera and laser pico-projector and is mounted on a user’s shoulder. But Harrison said the device ultimately could be the size of a deck of cards, or even a matchbox, so that it could fit in a pocket, be easily wearable, or be integrated into future handheld devices.

“With OmniTouch, we wanted to capitalize on the tremendous surface area the real world provides,” says Benko, a researcher in Microsoft Research’s Adaptive Systems and Interaction group. “We see this work as an evolutionary step in a larger effort at Microsoft Research to investigate the unconventional use of touch and gesture in devices to extend our vision of ubiquitous computing even further.

Harrison previously worked with Microsoft Research to develop Skinput, a technology that used bioacoustic sensors to detect finger taps on a person’s hands or forearm. Skinput thus enabled users to control smartphones or other compact computing devices.

The optical sensing used in OmniTouch, by contrast, allows a wide range of interactions, similar to the capabilities of a computer mouse or touchscreen. It can track three-dimensional motion on the hand or other commonplace surfaces, and can sense whether fingers are “clicked” or hovering.

What’s more, OmniTouch does not require calibration—users can simply wear the device and immediately use its features. No instrumentation of the environment is needed; only the wearable device is needed.

The work was funded by Microsoft.

More news from Carnegie Mellon University:

chat7 Comments


  1. Jill

    Skinput sounds interesting. What about a paint on surface that would recognize touch? Then you wouldn’t need a projector.

  2. Greg Madison

    I was a finalist in the Microsoft Imagine Cup with this project in 2009, just before the 6th Sense came out. Ironically, it is called the 7th Sense.The concept is very similar to mine!

  3. rob ver maas

    Greg, what you should do is advise the Patent Office that the novelty was recognised by the competition organiserss in their shortlisting you as a finalist. In other words, they admitted it didnt exist within their ranks beforehand. That move also is a type of ‘publishing’ of the work when the finalists were announced.

    These are the two essential ingredients for gaining a patent, regardless of any slight wording semantics they may want to play about features. Advising the PO now, or when MS lodges their patent knocks put their ownership of those facttors thereby preventing MS plagiarising your work.
    No expensive legal battle needed, just tell MS that you have already advised the PO and let them make an offer for you to withdraw it

  4. Amy

    then it is only a matter of time until we can translate sign language or gestures directly into wireless communication

  5. Wii U

    that’s pretty cool. We’re just a step away from those wicked holographic interfaces in the new Iron Man movies!

  6. Adidas PANDA shoes

    great informations

  7. albiz

    I think it’would change our life just like smartphones as they brought a great revolution in 21st century…. And now its time for another way of advancing….. i really like it….

We respect your privacy.