Wetzstein’s lab has created a 4d camera, expanding on Levoy and Hanrahan’s light field photography research

Professor Gordon Wetzstein, left; postdoctoral research fellow Donald Dansereau (Image credit: L.A. Cicero)
July 2017

A new 4D camera designed by Professor Gordon Wetzstein and postdoc Dr. Donald Dansereau captures light field information over a 138° field of view.

The difference between looking through a normal camera and the new design is like the difference between looking through a peephole and a window, the scientists said.

"A 2D photo is like a peephole because you can't move your head around to gain more information about depth, translucency or light scattering," Dansereau said. "Looking through a window, you can move and, as a result, identify features like shape, transparency and shininess."

That additional information comes from a type of photography called light field photography, first described in 1996 by EE Professors Marc Levoy and Pat Hanrahan. Light field photography captures the same image as a conventional 2D camera plus information about the direction and distance of the light hitting the lens, creating what's known as a 4D image. A well-known feature of light field photography is that it allows users to refocus images after they are taken because the images include information about the light position and direction. Robots might use this to see through rain and other things that could obscure their vision.

The extremely wide field of view, which encompasses nearly a third of the circle around the camera, comes from a specially designed spherical lens. However, this lens also produced a significant hurdle: how to translate a spherical image onto a flat sensor. Previous approaches to solving this problem had been heavy and error prone, but combining the optics and fabrication expertise of UCSD and the signal processing and algorithmic expertise of Wetzstein's lab resulted in a digital solution to this problem that not only leads to the creation of these extra-wide images but enhances them.

This camera system's wide field of view, detailed depth information and potential compact size are all desirable features for imaging systems incorporated in wearables, robotics, autonomous vehicles and augmented and virtual reality.

"Many research groups are looking at what we can do with light fields but no one has great cameras. We have off-the-shelf cameras that are designed for consumer photography," said Dansereau. "This is the first example I know of a light field camera built specifically for robotics and augmented reality. I'm stoked to put it into peoples' hands and to see what they can do with it."


Two 138° light field panoramas and a depth estimate of the second panorama. (Image credit: Stanford Computational Imaging Lab and Photonic Systems Integration Laboratory at UC San Diego)


Read more at Professor Wetztein's research site, Stanford Computational Imaging Lab.

Excerpted from Stanford News, "New camera designed by Stanford researchers could improve robot vision and virtual reality," July 21, 2017.