Talk Abstract:  Generating and transforming content requires both creativity and skill. While creativity is believed to be abundant, skill can often be a barrier to creativity. In our team, we aim to substantially reduce this barrier, empowering anyone into a creator. In this talk, I’ll present two streams of our work, we found necessary to simplify the creation process. First, we’ll focus on the creation, transformation and animation themselves. I’ll introduce a 3D generator that supports all ImageNet object categories. Then, we’ll look at a new 3D animation framework that is trained purely on monocular videos and capable of reconstructing shape, normals, color, and articulation parameters for a wide range of object categories. Then, I will discuss the concept of learnable game engines and show their applications in playing games and manipulating real-world videos of 3D games using text prompts, styles, and camera trajectories. In the second stream of work in our team we focus on making the creative process instantaneous. We’ll discuss our neural rendering approach, which works in real-time on a variety of devices and has a small hardware footprint. Finally, I’ll show our latest SnapFusion, which is currently the fastest text-to-image diffusion model that generates high-quality images on mobile devices in just 2 seconds.

Speaker Biography: Sergey Tulyakov is a Principal Research Scientist at Snap Inc. He leads the Creative Vision team and focuses on creating methods for transforming the world via computer vision and machine learning. His work includes 2D and 3D synthesis, photorealistic manipulation and animation, video synthesis, prediction and retargeting. Sergey pioneered the unsupervised image animation domain with MonkeyNet and First Order Motion Model that sparked a number of startups in the domain. His work on Interactive Video Stylization received the Best in Show Award at SIGGRAPH Real-Time Live! 2020. He has published 40+ top conference papers, journals and patents resulting in multiple innovative products, including Snapchat Pet Tracking, OurBaby, Real-time Neural Lenses, recent real-time try-on and many others. Before joining Snap Inc., Sergey was with Carnegie Mellon University, Microsoft, NVIDIA. He holds a PhD degree from the University of Trento, Italy.