SCIEN Talk

The technological advances in immersive telepresence are greatly impeded by the challenges that need to be met when mediating the realistic feeling of presence in a remote environment to a local human user. Providing a stereoscopic 360° visual representation of the distant scene further fosters the level of realism and greatly improves task performance. State-of-the-art technology has primarily developed catadioptric or multi-camera systems to address this issue. Current solutions are bulky, not realtime capable, and tend to produce erroneous image content due to the stitching processes involved, which are prone to perform poorly for texture-less scenes. In this talk, I will introduce a vision on-demand approach that creates stereoscopic scene information upon request. A real-time capable camera system along with a novel deep learning-based delay-compensation paradigm will be presented that provides instant visual feedback for highly immersive telepresence.

Traditional lenses are optimized for 2D imaging, which prevents them from capturing extra dimensions of the incident light field (e.g. depth or high-speed dynamics) without multiple exposures or moving parts. Leveraging ideas from compressed sensing, I replace the lens of a traditional camera with a single pseudorandom free-form optic called a diffuser. The diffuser creates a pseudorandom point spread function which multiplexes these extra dimensions into a single 2D exposure taken with a standard sensor. The image is then recovered by solving a sparsity-constrained inverse problem. This lensless camera, dubbed DiffuserCam, is capable of snapshot 3D imaging at video rates, encoding a high-speed video (>4,500 fps) into a single rolling-shutter exposure, and video-rate 3D imaging of fluorescence signals, such as neurons, in a device weighing under 3 grams.

In this talk, Dr. Jonghyun Kim will present two recent AR display prototypes inspired by human visual system. The first, Foveated AR dynamically provides high-resolution virtual image to the user's foveal region based on the tracked gaze. The second, Prescription AR is a prescription-embedded fully-customized AR display systems which works as user's eyeglasses and AR display at the same time. Finally, he will discuss on important issues for socially-acceptable AR display systems including customization, privacy, fashion, and eye-contact interaction, and how they are related to the display technologies.

AR on handheld, monocular, "through-the-camera" platforms such as mobile phones is a challenging task. While traditional, geometry based approaches provide useful data in certain scenarios, for truly immersive experiences we need to leverage the prior knowledge encapsulated in learned CNNs. In this talk I will discuss the capabilities and limitations of such traditional methods, the need for CNN-based solutions, and the challenges to training accurate and efficient CNNs on this task. I will describe our recent work on implicit, 3D representations for AR, with applications in novel view synthesis, scene reconstruction and arbitrary object manipulation. Finally, I will present a project opportunity, to learn such representations from a dataset of single images.

Traditionally, the image processing pipelines of consumer cameras have been carefully designed, hand-engineered systems. But treating an imaging pipeline as something to be learned instead of something to be engineered has the potential benefits of being faster, more accurate, and easier to tune. Relying on learning in this fashion presents a number of challenges, such as fidelity, fairness, and data collection, which can be addressed through careful consideration of neural network architectures as they relate to the physics of image formation. In this talk I'll be presenting recent work from Google's computational photography research team on using machine learning to replace traditional building blocks of a camera pipeline. I will present learning based solutions for the classic tasks of denoising, white balance, and tone mapping, each of which uses a bespoke ML architecture that is designed around the specific constraints and demands of each task. By designing learning-based solutions around the structure provided by optics and camera hardware, we are able to produce state-of-the-art solutions to these three tasks in terms of both accuracy and speed.

The use of imaging systems has grown enormously over the last several decades; these systems are an essential component in mobile communication, medicine, and automotive applications. As imaging applications have expanded the complexity of imaging systems hardware - from optics to electronics - has increased dramatically. The increased complexity makes software prototyping an essential tool for the design of novel systems and the evaluation of components. I will describe several simulations we created for image systems engineering applications: (a) designing cameras for autonomous vehicles [1], (b) simulating image encoding by the human eye and retina for image quality assessments [2], and (c) assessing the spatial sensitivity of CNNs for multiple applications [3]. This is a good moment to consider how academia and industry might cooperate to create an image systems simulation infrastructure that speeds the development of new systems for the many opportunities that will arise over the next few decades.

Each year the Stanford Center for Image Systems Engineering (SCIEN) holds an annual meeting for its' Industry Affiliate Member companies. The talks introduce new Stanford faculty who are advancing imaging science and technology. The poster presentations introduce postdoctoral researchers and graduate students who are working in computational imaging with expertise in image systems engineering, including optics, sensors, processing, machine learning, displays and human perception.

Please see the details on SCIEN website.

Light fields can be understood as the knowledge of the intensity and direction of each ray. But the light field does not contain the diffraction effects. The wavefront phase measurement does allow an approach to the measurement of these effects. The wavefront phase sensors are fundamental in any Adaptive Optics (AO) system, both in Astronomy and Ophthalmology. With AO it is possible to correct the images degraded by the atmospheric turbulence in large telescopes or sharply improve the retina images of the human eye. In fact, the wavefront phase is proportional to the refractive index changes, that is, it serves to detect structure in transparent objects. The same data acquired by a plenoptic sensor can be used to obtain the tomographic distribution of the atmosphere over the telescope, but also to detect transparent biological tissue, such as the neuronal one, in microscopes and without needing chemical markers. In this talk we will explain how to use the wavefront phase sensors as a complement to the acquisition of the light field, and we will show direct applications in fields as disparate as computational imaging (SEBI camera: real-time Full-HD single lens light field camera), silicon metrology (one-shot subnanometer accuracy on silicon wafers), and ophthalmology (wow!).

Demo following the talk: Wooptix presents the first light field video camera using a single adaptive optics lens; no resolution killing micro lens array. The light field video is processed live using a high end PC graphic card and displayed in HD resolution on a 3D display (no head worn glasses) using the 2D all-in-focus image coupled with a depth map generated using the focal stack. The demo consist of a small handheld video camera with adaptive optics, a PC for instant light field video processing, and a 3D display for live viewing of our light field video.

This presentation overviews fluorescence lifetime spectroscopy and imaging techniques for label-free in vivo characterization of biological tissues. Emphasis is placed on recently developed devices and methods enabling real-time characterization and diagnosis of diseased tissues during clinical interventions. I will present studies conducted in animal models and human patients demonstrating the ability of these techniques to provide rapid in-situ evaluation of tissue biochemistry and their potential to guide surgical and intravascular procedures. Current results demonstrate that intrinsic fluorescence can provide useful contrast for the diagnosis of vulnerable atherosclerotic plaques, intraoperative delineation of brain tumors and head and neck tumors. Finally, I will present results from the first-in-human study that shows the potential of a multispectral fluorescence lifetime method for image-guided augmented reality in trans-oral robotic surgery (TORS).

In many active optical systems where the light is supplied such as a range finder or optical radar (LiDAR), there is a possibility that the detector (pixel) can be destroyed if significant signal is returned or at the very least blinded for a period of time. This is because the designer struggles with the "one over range squared" loss which can amount to significant attenuation in the return signal given the range requirements. When pushing the range limit requirement, the sensor is in need of a large dynamic range detector and/or some form of detector protection when the target is quite close. This work proposes a lens that attempts to compensate for range signal loss passively and instantaneously by combining lens elements in parallel rather than in series as is typically done [Mudge Appl. Opt. 58, (2019)]. The proposed lens is relatively simple and compensates for range albeit not perfectly. Additionally, a discussion is provided to implement this approach along with a variety of examples of a range compensating lens [Phenis et al. Proc. of SPIE, 11125, (2019)]. These designs cover techniques and include some of the penalties incurred.