SCIEN Talk

Light fields can be understood as the knowledge of the intensity and direction of each ray. But the light field does not contain the diffraction effects. The wavefront phase measurement does allow an approach to the measurement of these effects. The wavefront phase sensors are fundamental in any Adaptive Optics (AO) system, both in Astronomy and Ophthalmology. With AO it is possible to correct the images degraded by the atmospheric turbulence in large telescopes or sharply improve the retina images of the human eye. In fact, the wavefront phase is proportional to the refractive index changes, that is, it serves to detect structure in transparent objects. The same data acquired by a plenoptic sensor can be used to obtain the tomographic distribution of the atmosphere over the telescope, but also to detect transparent biological tissue, such as the neuronal one, in microscopes and without needing chemical markers. In this talk we will explain how to use the wavefront phase sensors as a complement to the acquisition of the light field, and we will show direct applications in fields as disparate as computational imaging (SEBI camera: real-time Full-HD single lens light field camera), silicon metrology (one-shot subnanometer accuracy on silicon wafers), and ophthalmology (wow!).

Demo following the talk: Wooptix presents the first light field video camera using a single adaptive optics lens; no resolution killing micro lens array. The light field video is processed live using a high end PC graphic card and displayed in HD resolution on a 3D display (no head worn glasses) using the 2D all-in-focus image coupled with a depth map generated using the focal stack. The demo consist of a small handheld video camera with adaptive optics, a PC for instant light field video processing, and a 3D display for live viewing of our light field video.

This presentation overviews fluorescence lifetime spectroscopy and imaging techniques for label-free in vivo characterization of biological tissues. Emphasis is placed on recently developed devices and methods enabling real-time characterization and diagnosis of diseased tissues during clinical interventions. I will present studies conducted in animal models and human patients demonstrating the ability of these techniques to provide rapid in-situ evaluation of tissue biochemistry and their potential to guide surgical and intravascular procedures. Current results demonstrate that intrinsic fluorescence can provide useful contrast for the diagnosis of vulnerable atherosclerotic plaques, intraoperative delineation of brain tumors and head and neck tumors. Finally, I will present results from the first-in-human study that shows the potential of a multispectral fluorescence lifetime method for image-guided augmented reality in trans-oral robotic surgery (TORS).

In many active optical systems where the light is supplied such as a range finder or optical radar (LiDAR), there is a possibility that the detector (pixel) can be destroyed if significant signal is returned or at the very least blinded for a period of time. This is because the designer struggles with the "one over range squared" loss which can amount to significant attenuation in the return signal given the range requirements. When pushing the range limit requirement, the sensor is in need of a large dynamic range detector and/or some form of detector protection when the target is quite close. This work proposes a lens that attempts to compensate for range signal loss passively and instantaneously by combining lens elements in parallel rather than in series as is typically done [Mudge Appl. Opt. 58, (2019)]. The proposed lens is relatively simple and compensates for range albeit not perfectly. Additionally, a discussion is provided to implement this approach along with a variety of examples of a range compensating lens [Phenis et al. Proc. of SPIE, 11125, (2019)]. These designs cover techniques and include some of the penalties incurred.

The availability of academic and commercial light field camera systems has spurred significant research into the use of light fields and multi-view imagery in computer vision and computer graphics. In this talk, we discuss our results over the past few years, focusing on a few themes. First, we describe our work on a unified formulation of shape from light field cameras, combining cues such as defocus, correspondence, and shading. Then, we go beyond photoconsistency, addressing non-Lambertian objects, occlusions, and an SVBRDF-invariant shape recovery algorithm. Finally, we show that advances in machine learning can be used to interpolate light fields from very sparse angular samples, in the limit a single 2D image, and create light field videos from sparse temporal samples. We also discuss recent work on combining machine learning with plenoptic sampling theory to create virtual explorations of real scenes from a very sparse set of input images captured on a handheld mobile phone.

The Zeiss Group develops, produces and distributes measuring technology, microscopes, medical technology, eyeglass lenses, camera and cinema lenses, binoculars and semiconductor manufacturing equipment. In this talk, I will present a novel webcam based optical 3D scanning method that allows independent surface mesh generation inside X-ray microscopes. These surface models can be used for collision avoidance and improved ease of use.

Autonomous mobile robots (e.g., self-driving cars, delivery trucks) are emerging and reshaping the world. However, there is one critical problem blocking the entire industry: how can mobile robots drive safely and naturally in the complex 3D world, and how do we know? In this talk, I will discuss our solution, called "training academy", to the aforementioned problem. We apply 3D vision, deep learning, and reinforcement learning techniques to generate real-world, high-fidelity driving scenarios and train the autonomous systems to develop human-like intelligence in a simulated environment.

Rendering refers to a process of creating digital images of an object or a scene from 3D data using computers and algorithms. Inverse rendering is the inverse process of rendering, i.e., reconstructing 3D data from 2D images. The 3D data to be recovered can be 3D geometry, reflectance of a surface, camera viewpoints, or lighting conditions.
In this talk, we will discuss three inverse rendering problems. First, inverse rendering using flash photography captures 3D geometry and reflectance of a static object using a single camera and a flashlight attached to the camera. An alternating and iterative optimization framework is proposed to jointly solve for several unknown properties. Second, inverse rendering at microscale reconstructs 3D normals and reflectance of a surface at microscale. A specially designed acquisition system, as well as an inverse rendering algorithm for microscale material appearance, are proposed. Lastly, inverse rendering for human hair describes a novel 3D reconstruction algorithm for modeling high-quality human hair geometry. We hope that our work on these advanced inverse rendering problems boosts hyper-realism in computer graphics

Observing large-scale three-dimensional subcellular dynamics in vivo at high spatiotemporal resolution has long been a pursuit for biology. However, both the signal-to-noise ratio and resolution degradation in multicellular organisms pose great challenges. In this talk, I will discuss our recent work in in vivo aberration-free 3D fluorescence imaging at millisecond scale by scanning light-field microscopy with digital adaptive optics. Specifically, we propose scanning light-field microscopy to achieve diffraction-limited 3D synthetic aperture for incoherent conditions, which facilitates real-time digital adaptive optics for every pixel in post-processing. Various fast subcellular processes are observed, including mitochondrial dynamics in cultured neurons, membrane dynamics in zebrafish embryos, and calcium propagations in cardiac cells, human cerebral organoids, and Drosophila larval neurons, enabling simultaneous in vivo studies of morphological and functional dynamics in 3D.

In this talk, I will present an overview of our most recent advancements in Robotic Imaging, Machine Leaning and Medical Augmented Reality. I will first discuss the particular requirements for intra-operative imaging and visualization. I will then present some of our latest results in intra-operative multimodal robotic imaging and its translation to clinical applications. I will then discuss the impact of research advancement in machine learning on medical imaging and computer assisted intervention. I will finally present some applications of virtual and augmented reality in the medical domain. Starting by the current deployment of AR and VR technology within medical education, I discuss its current and future impact on surgical education and training. I will then review the first deployment of augmented reality into operating rooms in the last two decades and present some of our latest achievements in this field (see also: www.medicalaugmentedreality.org.

Combining photography and spectroscopy, spectral imaging enables us to see what no traditional color camera has seen before. The current trend is to miniaturize the technology and bring it towards industry. In this talk, I will first give a general introduction to the most common pitfalls of spectral imaging and the challenges that come with miniaturization. Major pitfalls include balancing cross-talk, quantum efficiency, illumination and the optics. Miniaturization has become possible thanks to the monolithic per-pixel integration of thin-film Fabry-Pérot filters on CMOS imaging sensors. I will explain the difficulty of using these cameras with non- telecentric lenses. This is a major concern because of the angular dependency of the thin-film filters. I will demonstrate how this important issue can be solved using a model-based approach.