SAND: Enhancing Open-Set Neuron Descriptions through Spatial Awareness

We propose Spatially-Aware open-set Network Dissection (SAND) a technique to identify and label the learned representation of the neurons of deep vision networks. Contrary to earlier open-vocabulary neuron explanation methods we also leverage a neuron's spatial pattern of activation to guide our predictions towards more accurate and relevant concepts while avoiding being misled by confounding visual information. We highlight important regions for a neuron through image masking which has the advantage of being able to block out irrelevant concepts from an image handling irregularly shaped activation regions and revealing the visual concepts that a neuron learns in order to identify objects. We use CLIP to connect highly activating image regions with descriptive concepts and measure the quality of our results through human evaluation. Further since such manual evaluation can be highly time consuming costly and unscalable we also propose an automated approach which uses image generation to get quantitative feedback on the generated concepts. Finally as an application of our interpretability method we demonstrate how it can be tuned to the medical domain. Our code is available at https://github.com/Trustworthy-ML-Lab/SAND.