Modality-Incremental Learning with Disjoint Relevance Mapping Networks for Image-Based Semantic Segmentation

In autonomous driving environment perception has significantly advanced with the utilization of deep learning techniques for diverse sensors such as cameras depth sensors or infrared sensors. The diversity in the sensor stack increases the safety and contributes to robustness against adverse weather and lighting conditions. However the variance in data acquired from different sensors poses challenges. In the context of continual learning (CL) incremental learning is especially challenging for considerably large domain shifts e.g. different sensor modalities. This amplifies the problem of catastrophic forgetting. To address this issue we formulate the concept of modality-incremental learning and examine its necessity by contrasting it with existing incremental learning paradigms. We propose the use of a modified Relevance Mapping Network (RMN) to incrementally learn new modalities while preserving performance on previously learned modalities in which relevance maps are disjoint. Experimental results demonstrate that the prevention of shared connections in this approach helps alleviate the problem of forgetting within the constraints of a strict continual learning framework.