Unsupervised Supervised Learning I: Estimating Classification and Regression Errors without Labels

Estimating the error rates of classifiers or regression models is a fundamental task in machine learning which has thus far been studied exclusively using supervised learning techniques. We propose a novel unsupervised framework for estimating these error rates using only unlabeled data and mild assumptions. We prove consistency results for the framework and demonstrate its practical applicability on both synthetic and real world data. [abs] [ pdf ][ bib ] © JMLR 2010. (edit, beta)