SilverLining: Data-First Mitigation of Spatial and Spectral Shortcuts Without Introducing New Confounders

Deep neural networks exploit shortcuts--spurious correlations like laterality markers (spatial) or scanner-specific noise (spectral)--that severely compromise generalization. Many healthcare applications face multiple concurrent shortcuts that are both spatial and spectral, which existing methods struggle to handle. We present SilverLining, an attention-based preprocessing framework that simultaneously identifies and mitigates both spatial and spectral shortcuts without introducing new spurious correlations. Our key insight is that naive removal of shortcut features can itself create new shortcuts, where models learn to exploit the removal patterns as new spurious correlations. We address this through a principled confounder-free correction strategy that maintains consistent preprocessing patterns across all classes in both spatial and frequency domains, preventing new confounders. Extensive experiments demonstrate SilverLining's effectiveness: achieving 0.87 AUC on controlled vision tasks and 0.94 AUC on counter-shortcut medical imaging evaluation where shortcuts are reversed; improving cross-institutional chest X-ray classification from 0.72 to 0.77 AUC; and 0.54 mAP on polyp detection despite natural spurious correlations from surgical overlays. Our data-centric approach provides an effective solution for reducing multiple types of data shortcuts without architectural modifications, creating preprocessed datasets that improve model robustness across both classification and detection tasks. Our codebase is available at https://github.com/theidentity/SilverLining_WACV2026/.