Differentiable Rendering for Endoscopic Scene Reconstruction

Intraoperative reconstruction is an immensely challenging task, where conventional reconstruction algorithms would quickly fail in intraoperative applications. This work takes a different perspective on the localization and model reconstruction for intraoperative applications. Therefore, a fully differentiable framework for intraoperative localization, and geometry and texture reconstruction is presented.In addition, a differentiable re-projection process is presented to map 2D information to a 3D mesh representation.For this intraoperative reconstruction, various types of simultaneous localization and mapping (SLAM) algorithms are conventionally used in the literature for scene reconstruction problems. In contrast, this work addresses intraoperative scene reconstruction with differentiable rendering techniques. A parameterized mesh is used for model representation, where parameters are adjusted such that the rendered image resembles the observed image data. The work benefits from the latest methods in the field of differential rendering. However, while available differentiable rendering methods in the literature cover functions for mapping 3D sceneries onto an image plane, they do not consider the inverse step of mapping from the image plane back to the mesh model. This work aims to bridge this gap in the literature by also presenting a inverse rendering  process.