Wearable cameras film actors move in the wild

CARNEGIE MELLON U. (US) — A system of body-mounted cameras can capture motion in almost any environment for use in digital animation.

Unlike traditional motion capture techniques, which use cameras to meticulously record the movements of actors inside studios, the new system turns the cameras around—mounting almost two dozen, outward-facing cameras on the actors themselves. The motion capture can occur almost anywhere—in natural environments, over large areas, and outdoors.


Motion capture makes possible scenes such as those in the film Pirates of the Caribbean: Dead Man’s Chest, where the movements of actor Bill Nighy were translated into a digitally created Davy Jones with octopus-like tentacles forming his beard. But body-mounted cameras enable capture of motions, such as running outside or swinging on monkey bars, that would be difficult—if not impossible—otherwise, says Takaaki Shiratori, a post-doctoral associate at Disney Research, Pittsburgh (DRP).

“This could be the future of motion capture,” says Shiratori, who made a presentation about the new technique this month at SIGGRAPH 2011, the International Conference on Computer Graphics and Interactive Techniques in Vancouver. As video cameras become ever smaller and cheaper, “I think anyone will be able to do motion capture in the not-so-distant future,” he adds.

The wearable camera system, developed by a team of researchers at Carnegie Mellon University and DRP, makes it possible to reconstruct the relative and global motions of an actor thanks to a process called structure from motion (SfM). Takeo Kanade, a professor of computer science and robotics at Carnegie Mellon University, developed SfM 20 years ago as a means of determining the three-dimensional structure of an object by analyzing the images from a camera as it moves around the object, or as the object moves past the camera.

In this application, SfM is not used primarily to analyze objects in a person’s surroundings, but to estimate the pose of the cameras on the person. Researchers used Velcro to mount 20 lightweight cameras on the limbs, and trunk of each subject. Each camera was calibrated with respect to a reference structure. Each person then performed a range-of-motion exercise that allowed the system to automatically build a digital skeleton and estimate positions of cameras with respect to that skeleton.

SfM is used to estimate rough position and orientation of limbs as the actor moves through an environment and to collect sparse 3D information about the environment that can provide context for the captured motion. The rough position and orientation of limbs serves as an initial guess for a refinement step that optimizes the configuration of the body and its location in the environment, resulting in the final motion capture result.

The quality of motion capture from body-mounted cameras does not yet match the fidelity of traditional motion capture, Shiratori says, but will improve as the resolution of small video cameras continues to improve.

The technique requires a significant amount of computational power; a minute of motion capture now can require an entire day to process. Future work will include efforts to find computational shortcuts, such as performing many of the steps simultaneously through parallel processing.

More news from Carnegie Mellon University: www.cmu.edu/news