← Back to papers
2019 NIPS NeurIPS 2019

Multiclass Performance Metric Elicitation

Abstract

Metric Elicitation is a principled framework for selecting the performance metric that best reflects implicit user preferences. However, available strategies have so far been limited to binary classification. In this paper, we propose novel strategies for eliciting multiclass classification performance metrics using only relative preference feedback. We also show that the strategies are robust to both finite sample and feedback noise.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Machine Learning
🧭 Keyword Pioneer — metric elicitation
🐣 Hot Topic Early Bird — multiclass classification
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Gaurush Hiranandani , Shant Boodaghians , Ruta Mehta , Oluwasanmi O Koyejo

Topics

Artificial Intelligence > Core AI > Causal Inference Machine Learning > Core Methods > Classification Machine Learning > Learning Types > Classification Machine Learning > Learning Types > Metric Learning Machine Learning > Core Methods > Evaluation

Keywords

preference learning multiclass classification performance metric preference feedback metric elicitation
Download PDF

Related papers

Two Generator Game: Learning to Sample via Linear Goodness-of-Fit Test 2019
Metalearned Neural Memory 2019
Model Similarity Mitigates Test Set Overuse 2019
Continual Unsupervised Representation Learning 2019
Reinforcement Learning with Convex Constraints 2019