EndoPBR: Photorealistic Synthetic Data for Surgical 3D Vision via Physically-based Rendering

Synthetic data has played a pivotal role in developing large-scale 3D vision models due to its high-quality annotations and ease of curation. In domains where labeled data collection is difficult, such as endoscopy, synthetic data holds promise as a means to generate the large-scale annotated datasets required to train modern neural networks. In this work, we address a core question for data-scarce applications in 3D vision: how can we generate synthetic labeled data, and how useful would the data be for training downstream vision models? First, we introduce a novel data generation module that takes images with known geometry and camera poses as input and estimates the material and lighting conditions of the scene. To stabilize training, we leverage domain-specific properties like non-stationary lighting and anatomical material priors. We model the material properties as a bidirectional reflectance distribution function, parameterized by a neural network. Via the rendering equation, we can generate photorealistic images at arbitrary camera poses. We demonstrate that this method produces competitive novel view synthesis results compared to previous work while being more lightweight, flexible, and efficient. Second, we use our synthetic data to train models on various downstream 3D vision tasks and find that models trained solely on our synthetic data generally outperform those trained on real data across various metrics and tasks. Our experiments show that synthetic data is a promising avenue towards robust 3D vision in surgical scenes.