A Multi-Agent Diffusion Approach for MRI Anomaly Segmentation via Modality-Specific LoRA Specialization

Unsupervised anomaly segmentation in multi-sequence MRI is a promising way to scale lesion screening, but existing reconstruction-based methods face three persistent issues: they fail to generalize across modalities, they depend on hand-crafted masking or paired translations, and they often require separate models with high inference cost. In this work, we take a stepwise approach to address these limitations. In the first stage, we fully fine-tune a diffusion model on healthy brain MRI slices pooled across T1, T2, and FLAIR, which produces anatomically consistent reconstructions. To further improve, we introduce a lightweight second stage where modality-specific LoRA adapters are trained on top of the pretrained diffusion backbone. A simple router automatically selects the right adapter for each input, effectively turning the system into a modality-aware, agentic-like framework. To further stabilize reconstructions, we incorporate a learnable latent-frequency mask that suppresses non-informative spectral components and preserves structural detail. This design allows the model to emphasize healthy anatomy while efficiently capturing modality-dependent contrasts. This two-stage strategy boosts Dice to 88% on BraTS2021 (FLAIR), achieving state-of-the-art performance. Experiments on BraTS2021, ISLES, and ATLAS datasets confirm that the approach consistently improves Dice and SSIM across all modalities, outperforming diffusion, masking, and cycle-based baselines, and offering a practical balance between accuracy and efficiency for clinical MRI anomaly detection. Source code is available at https://github.com/wafaAlghallabi/MRI-Router.