Rethinking Cluster-Conditioned Diffusion Models for Label-Free Image Synthesis

Diffusion-based image generation models can enhance image quality when conditioned on ground truth labels. Here we conduct a comprehensive experimental study on image-level conditioning for diffusion models using cluster assignments. We investigate how individual clustering determinants such as the number of clusters and the clustering method impact image synthesis across three different datasets. Given the optimal number of clusters with respect to image synthesis we show that cluster-conditioning can achieve state-of-the-art performance with an FID of 1.67 for CIFAR10 and 2.17 for CIFAR100 along with a strong increase in training sample efficiency. We further propose a novel empirical method to estimate an upper bound for the optimal number of clusters. Unlike existing approaches we find no significant association between clustering performance and the corresponding cluster-conditional FID scores. Code is available at https://github.com/ HHU-MMBS/cedm-official-wavc2025