2025 AISTATS AISTATS 2025

$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization