SCIEN Talk

This talk will review architectures for computational imaging arrays where algorithms and cameras are co-designed. The talk will focus on novel digital readout integrated circuits (DROICs) that achieve snapshot on-chip high dynamic range and object tracking where most commercial systems require a multiple exposure acquisition.

The Stanford Center for Image Systems Engineering (SCIEN) is a partnership between the Stanford School of Engineering and technology companies developing imaging systems for the enhancement of human communication.

Active illumination techniques enable self-driving cars to detect and avoid hazards, optical microscopes to see deep into volumetric specimens, and light stages to digitally capture the shape and appearance of subjects. These active techniques work by using controllable lights to emit structured illumination patterns into an environment, and sensors to detect and process the light reflected back in response. Although such techniques confer many unique imaging capabilities, they often require long acquisition and processing times, rely on predictive models for the way light interacts with a scene, and cease to function when exposed to bright ambient sunlight.

In this talk, we introduce a generalized form of active illumination—known as optical probing—that provides a user with unprecedented control over which light paths contribute to a photo. The key idea is to project a sequence of illumination patterns onto a scene, while simultaneously using a second sequence of mask patterns to physically block the light received at select sensor pixels. This all-optical technique enables RAW photos to be captured in which specific light paths are blocked, attenuated, or enhanced. We demonstrate experimental probing prototypes with the ability to (1) record live direct-only or indirect-only video streams of a scene, (2) capture the 3D shape of objects in the presence of complex transport properties and strong ambient illumination, and (3) overcome the multi-path interference problem associated with time-of-flight sensors.

Around a year ago we set out to create an open-source reference design for a 3D-360 camera. In nine months, we had designed and built the camera, and published the specs and code. Our team leveraged a series of maturing technologies in this effort. Advances and availability in sensor technology, 20+ of computer vision algorithm development, 3D printing, rapid design photo-typing and computation photography allowed our team to move extremely fast. We will delve into the roles each of these technologies played in the designing of the camera, giving an overview of the system components and discussing the tradeoffs made during the design process. The engineering complexities and technical elements of 360 stereoscopic video capture will be discussed as well. We will end with some demos of the system and its output.

The ultimate wearable display is an information device that people can use all day. It should be as forgettable as a pair of glasses or a watch, but more useful than a smart phone. It should be small, light, low-power, high-resolution and have a large field of view (FOV). Oh, and one more thing, it should be able to switch from VR to AR.

These requirements pose challenges for hardware and, most importantly, optical design. In this talk, I will review existing AR and VR optical architectures and explain why it is difficult to create a small, light and high-resolution display that has a wide FOV. Because comfort is king, new optical designs for the next-generation AR and VR system should be guided by an understanding of the capabilities and limitations of the human visual system.

From UCLA Shannon Centennial Celebration website:

Claude Shannon was an American mathematician, electrical engineer, and cryptographer known as "the father of information theory". Shannon founded information theory and is perhaps equally well known for founding both digital computer and digital circuit design theory. Shannon also laid the foundations of cryptography and did basic work on code breaking and secure telecommunications.

Events taking place around the world are listed at IEEE Information Theory Society.

The recent progress in recognizing visual objects and annotating images has been driven by super-rich models and massive datasets. However, machine vision models still have a very limited 'understanding' of images, rendering them brittle when attempting to generalize to unseen examples. I will describe recent efforts to improve the robustness and accuracy of systems for annotating and retrieving images, first, by using structure in the space of images and fusing various types of information about image labels, and second, by matching structures in visual scenes to structures in their corresponding language descriptions or queries. We apply these approaches to billions of queries and images, to improve search and annotation of public images and personal photos.

Many creative ideas are being proposed for image sensor designs, and these may be useful in applications ranging from consumer photography to computer vision. To understand and evaluate each new design, we must create a corresponding image-processing pipeline that transforms the sensor data into a form that is appropriate for the application. The need to design and optimize these pipelines is time-consuming and costly. I explain a method that combines machine learning and image systems simulation that automates the pipeline design. The approach is based on a new way of thinking of the image-processing pipeline as a large collection of local linear filters. Finally, I illustrate how the method has been used to design pipelines for consumer photography and mobile imaging.

The Camera Culture Group at the MIT Media Lab aims to create a new class of imaging platforms. This talk will discuss three tracks of research: femto photography, retinal imaging, and 3D displays.
Femto Photography consists of femtosecond laser illumination, picosecond-accurate detectors and mathematical reconstruction techniques allowing researchers to visualize propagation of light. Direct recording of reflected or scattered light at such a frame rate with sufficient brightness is nearly impossible. Using an indirect 'stroboscopic' method that records millions of repeated measurements by careful scanning in time and viewpoints we can rearrange the data to create a 'movie' of a nanosecond long event. Femto photography and a new generation of nano-photography (using ToF cameras) allow powerful inference with computer vision in presence of scattering.

EyeNetra is a mobile phone attachment that allows users to test their own eyesight. The device reveals corrective measures thus bringing vision to billions of people who would not have had access otherwise. Another project, eyeMITRA, is a mobile retinal imaging solution that brings retinal exams to the realm of routine care, by lowering the cost of the imaging device to a 10th of its current cost and integrating the device with image analysis software and predictive analytics. This provides early detection of Diabetic Retinopathy that can change the arc of growth of the world's largest cause of blindness.

Finally the talk will describe novel lightfield cameras and lightfield displays that require a compressive optical architecture to deal with high bandwidth requirements of 4D signals.

We present a new microscopy technique for 3D functional neuroimaging in live brain tissue. The device is a simple light field fluorescence microscope allowing full volume acquisition in a single shot and can be miniaturized into a portable implant. Our computational methods first rely on spatial and temporal sparsity of fluorescence signals to identify and precisely localize neurons. We compute for each neuron a unique pattern, the light-field signature, that accounts for the effects of optical scattering and aberrations. The technique then yields a precise localization of active neurons and enables quantitative measurement of fluorescence with individual neuron spatial resolution and at high speeds, all without ever reconstructing a volume image. Experimental results are shown on live Zebrafish.

More Information: www.nicolaspegard.com

Historically, virtual reality (VR) with head-mounted displays (HMDs) is associated with computer-generated content and gaming applications. However, recent advances in 360 degree cameras facilitate omnidirectional capture of real-world environments to create content to be viewed on HMDs - a technology referred to as cinematic VR. This can be used to immerse the user, for instance, in a concert or sports event. The main focus of this talk will be on data representations for creating such immersive experiences.

In cinematic VR, videos are usually represented in a spherical format to account for all viewing directions. To achieve high-quality streaming of such videos to millions of users, it is crucial to consider efficient representations for this type of data, in order to maximize compression efficiency under resource constraints, such as the number of pixels and bitrate. We formulate the choice of representation as a multi-dimensional, multiple choice knapsack problem and show that the resulting representations adapt well to varying content.

Existing cinematic VR systems update the viewports according to head rotation, but do not support head translation or focus cues. We propose a new 3D video representation, referred to as depth augmented stereo panorama, to address this issue. We show that this representation can successfully induce head-motion parallax in a predefined operating range, as well as generate light fields across the observer's pupils, suitable for using with emerging light field HMDs.