Many aspects of hospital care ranging from performing surgery to patient care in the ICU are complex and have significant impact on patient outcomes, yet data for studying the execution of these care pathways is typically sparse and not amenable to data-driven analyses. A major challenge has been the lack of a scalable means to collect rich data describing care execution. In this talk I will describe the potential of using computer vision interpretation of video captured in hospital environments to provide this source of data. I will discuss computer vision based observational study of hospital care processes in two environments, the operating room and the ICU. I will present key technical challenges of interpreting human behavior from video capture, and describe ongoing work on approaches for tackling them.