Separating Direct and Global Components from Novel Viewpoints

Separating an image of a scene illuminated by a light source into direct components such as specular reflection and diffuse reflection and global components such as inter-reflection and subsurface scattering is important as preprocessing for various computer vision and graphics applications. Conventional methods cannot separate direct and global components from novel viewpoints and have difficulties in robustly separating those components from a small number of images even from known viewpoints. In this paper we propose a method for synthesizing the direct and global components of a scene from novel viewpoints by using a relatively small number of images. Specifically our proposed method uses the multi-view images captured by using a coaxial projector-camera system and then recovers the density and radiance values of each component on the basis of neural radiance fields (NeRF). We conduct a number of experiments using real images captured with a projector-camera system and confirm the effectiveness of our method. In addition we demonstrate that our method is useful for two applications: image-based material editing and 3D shape recovery.