Engineers are using stripes to make images that regular cameras could never capture.
Their compact Hyperspectral Stripe Projector (HSP) is a step toward a new method to collect the spatial and spectral information required for self-driving cars, machine vision, crop monitoring, surface wear and corrosion detection, and other uses.
“I can envision this technology in the hands of a farmer, or on a drone, to look at a field and see not only the nutrients and water content of plants but also, because of the 3D aspect, the height of the crops,” says Kevin Kelly, an associate professor of electrical and computer engineering at Rice University’s Brown School of Engineering. “Or perhaps it can look at a painting and see the surface colors and texture in detail, but with near-infrared also see underneath to the canvas.”
Engineers turned a set of monochrome camera captures from a 3D hyperspectral imaging system based on the Hyperspectral Stripe Projector into 4D data of spatial and spectral information. This video demonstrates the spatial reconstruction of the targets. (Credit: Kelly Lab/Rice)
Kelly’s lab could enable 3D spectroscopy on the fly with a system that combines the HSP, a monochrome sensor array, and sophisticated programming to give users a more complete picture of an object’s shape and composition.
“We’re getting four-dimensional information from an image, three spatial and one spectral, in real time,” Kelly says. “Other people use multiple modulators and thus require bright light sources to accomplish this, but we found we could do it with a light source of normal brightness and some clever optics.”
HSP takes a cue from portable 3D imaging techniques that are already in people’s hands—think of face ID systems in smartphones and body trackers in gaming systems—and adds a way to pull broad spectral data from every pixel captured. This compressed data is reconstructed into a 3D map with spectral information that can incorporate hundreds of colors and be used to reveal not only the shape of an object but also its material composition.
“Regular RGB (red, green, blue) cameras basically give you only three spectral channels,” says Yibo Xu, lead author of the paper in Optics Express. “But a hyperspectral camera gives us spectra in many, many channels. We can capture red at around 700 nanometers and blue at around 400 nanometers, but we can also have bandwidths at every few nanometers or less between. That gives us fine spectral resolution and a fuller understanding of the scene.
“HSP simultaneously encodes the depth and hyperspectral measurements in a very simple and efficient way, allowing the use of a monochrome camera instead of an expensive hyperspectral camera as typically used in similar systems,” says Xu, who earned her doctorate at Rice in 2019 and is now a machine learning and computer vision research engineer at Samsung Research America Inc. She developed both the hardware and reconstruction software as part of her thesis in Kelly’s lab.
HSP uses an off-the-shelf digital micromirror device (DMD) to project patterned stripes that look something like colorful bar codes onto a surface. Sending the white-light projection through a diffraction grating separates the overlapping patterns into colors. Each color is reflected back to the monochrome camera, which assigns a numerical grey level to that pixel.
Each pixel can have multiple levels, one for every color stripe it reflects. These are recombined into an overall spectral value for that part of the object.
“We use a single DMD and a single grating in HSP,” Xu says. “The novel optical design of folding the light path back to the same diffraction grating and lens is what makes it really compact. The single DMD allows us to keep the light we want and throw away the rest.”
These finely tuned spectra can reach beyond visible light. What they reflect back to the sensor as multiplexed fine-band spectra can be used to identify the material’s chemical composition.
At the same time, distortions in the pattern are reconstructed into 3D point clouds, essentially a picture of the target, but with a lot more data than a plain snapshot could provide.
Kelly envisions HSP built into car headlights that can see the difference between an object and a person. “It could never get confused between a green dress and a green plant, because everything has its own spectral signature,” he says.
He believes the lab will eventually incorporate ideas from Rice’s single-pixel camera to further reduce the size of the device and adapt it for compressive video capture as well.
The National Science Foundation funded the research.
Source: Rice University
